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ABSTRACT 


The  validity  of  the  Display  Evaluative  Index  technique  was  investi¬ 
gated  in  two  separate  studies.  Both  studies  indicated  adequate  predictive 
validity  for  the  technique.  The  ability  of  the  technique  to  predict  "decision- 
making"  was  found  to  be  superior  to  its  ability  to  predict  "information  ex¬ 
traction"  processes.  Generally,  it  was  found  that  pictorial  type  displays 
are  superior  to  tabular  displays  for  "decision-making";  no  difference 
seemed  manifest  between  the  two  display  types  for  "information  extraction.  " 
Adding  appropriate  color  coding  seemed  to  augment  the  ability  of  the  user 
to  employ  display  information.  As  fact  density  increased,  display  decision¬ 
making  scores  decreased. 
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EVALUATION 


This  study  is  one  of  several  exploratory  investigations 
directed  to  the  perplexing  and  difficult  problem  of  pre¬ 
dicting  the  operational  effectiveness  of  alternative  or  com¬ 
peting  large-scale  display  designs.  As  one  approach  to  the 
problem,  the  console-oriented  Display  Evaluative  Index  (DEI) 
vas  modified  by  its  originators  to  provide  a  figure  of  merit 
based  on  decision  adequacy  rather  than  control  action  per¬ 
formance.  The  approach  is  consistent  vit.h  our  view  that 
decision-making  facilitation  is  the  most  important  objective 
of  good  large-scale  display  designs. 

Outstanding  among  several  contributions  made  during  the 
course  of  this  investigation  are: 

a.  The  modified  DEI  shows  considerable  promise  of  being 
the  most  effective  means  presently  available  for  assessing 
and  predicting  display  performer ?e .  it  should  be  noted,  how¬ 
ever,  that  application  of  such  analytic  procedures  as  the  DEI 
must  be  limited  to  situations  where  the  utilization  require¬ 
ments  for  displayed  information  are  veil  defined. 

b.  Within  the  limitations  cf  tl.c  experimental  design, 
the  data  support  the  contention  that  information  utilization, 
rather  than  information  extraction,  provides  the  most  appro¬ 
priate  criterion  for  measuring  operational  effectiveness. 

Encouraged  by  the  results  of  this  initial  study,  ve 
plan  to  continue  work  toward  refinement  of  the  DEI  technique. 
As  part  of  the  proposed  study,  a  list  of  utilization  factors 
will  be  prepared,  together  with  new  design  principles,  for  the 
organization  and  presentation  of  displayed  information.  Per¬ 
formance  tests  will  be  conducted  to  determine  the  capability 
of  the  improved  DEI  to  provide  the  link  between  decision  ade¬ 
quacy  and  individual  characteristics  of  display  design. 


u. 

WILLIAM  J 


Project  Engineer 
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CHAPTER  I 


INTRODUCTION 


Developers  and  users  of  large-scale  displays  possess  the  require¬ 
ment  for  evaluating  these  displays,  from  the  information  transfer  point  of 
view,  while  the  displays  are  in  the  design  proposal  stages,  Aside  from 
various  subjective  evaluations,  which  have  been  and  are  being  used  exten¬ 
sively  for  evaluating  displays,  information  presentation  has  also  been 
analyzed  previously  from  the  points  of  view  of  individual  parameters.  For 
example,  studies  have  been  performed  in  such  areas  as  character  legibility, 
form,  size,  color,  and  information  density.  However,  few  techniques  are 
available  which  employ  objective  measurement  and  calculational  methods 
for  evaluating  large-scale  displays. 

The  values  of  such  an  objective  technique  would  be  manyfold.  Costly 
design  errors  might  be  avoided  or  minimized,  the  effectiveness  of  alterna¬ 
tive  display  designs  might  be  determined  early  in  the  development  stages, 
and  performance  t  stimates  for  the  man-machine  integral  could  be  derived 
while  it  is  early  enough  to  correct  a  failure  to  achieve  system  requirements. 


The  Display  Evaluative  Index  (DEI) 

Applied  Psychological  Services  has  developed  a  technique  for  quan¬ 
titatively  evaluating  the  ability  of  displays  to  transfer  information  to  opera¬ 
tors  in  a  syatem,and  for  the  operators  to  act  on  this  information  (Siegel, 
Miehle,  and  Federman,  1964).  The  technique  is  called  the  Display  Evalua¬ 
tive  Index  (DEI)  technique  and  yields  a  number  which  is  said  to  represent  a 
figure  of  merit  for  a  given  display.  The  DEI  technique  was  drawn  from  a 
number  of  information  transfer  principles  and  has  been  applied  to  a  number 
and  range  of  military  equipments,  including  command  and  control  systems. 
The  numerical  value  resulting  from  application  of  the  DEI  technique  to  a 
display  is  based  on  identifiable  and  measurable  parameters.  The  technique, 
which  has  been  validated,  does  not  require  experimental  verification  each 
time  it  is  used,  Freitag  (1966),  in  reviewing  the  literature  on  man-machine 
hardware  operability  assessment,  stated  that  where  information  transfer 
efficiency  between  a  system  and  a  human  operator  is  concerned,  an  index 
such  as  the  DEI  may  be  all  that  is  required.  Specifically,  Freitag  main¬ 
tained  that  ",  .  .  of  the  presently  available  systematic  attempts  at  definition, 
the  most  promising  techniques  were  determined  to  be  those  utilizing  infor¬ 
mation  transfer  concepts  (Display  Evaluative  Index),  empirical  measurements 
(the  s/n  ratio),  and  the  computerized  technique  (Monte  Carlo  and  man-in-the- 
loop  simulation).  " 
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Principles  Incorporated  into  DEI 


In  order  to  provide  an  index  which  possesses  concept  as  well  as 
empirical  validity,  the  DEI  was  made  to  reflect  a  number  of  information 
transfer  principles.  The  following  conceptual  principles  are  incorporated 
into  the  DEI  technique.  All  else  being  equal,  that  system  is  best  which: 

1.  requires  the  least  operator  information  pro¬ 
cessing  per  subtask  unit.  For  example,  a 
predictive  display  is  considered  superior  to 
a  non-predictive  display. 

2.  has  the  greatest  directness  between  the  in¬ 
formation  transmitters  and  the  decisions. 

For  example,  as  the  number  of  nodes  in  a 
communications  network  increases,  the  ef¬ 
ficiency  of  the  network  decreases  and  its 
vulnerability  increases. 

3.  has  the  least  difference  between  the  amount 
of  information  presented  and  that  required. 

For  example,  differential  operator  filtering 
and/or  amplification  may  serve  to  increase 
error  potential  and  decrease  information 
transfer  effectiveness. 

4.  provides  for  redundancy  of  information. 

For  example,  a  message  can  be  decoded 
quicker  and  with  fewer  errors  when  it  con¬ 
tains  redundant  information. 

5.  requires  the  least  intermediate  data  pro¬ 
cessing  by  the  operator  before  he  can  reach 
a  decision.  For  example,  unit  conversions, 
transformations  and  the  like,  serve  to  delay 
information  transfer  and  to  introduce  error 
potential. 

6.  has  the  least  number  of  information  sources. 

For  example,  Kaufman  and  Kaufman  (I960) 
suggest  that  the  error  potential  of  a  system 
increases  as  its  size  and  volume  (number  of 
components)  increases. 

The  aspects,  enumerated  above,  are  represented  by  factors  which  vary 
from  zp'O  to  one.  A  factor  has  the  value  of  one  when  the  system  is  "perfect" 
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with  respect  to  that  factor.  The  DEI  is  the  product  of  all  factors  involved, 
each  suitably  weighted  by  means  of  an  exponent.  The  results  of  the  multi¬ 
plication  is  that  the  DEI  also  varies  between  zero  and  one.  The  implication 
of  the  multiplicative  form  is  that  "a  chain  is  as  strong  as  its  weakest  link.  " 
An  alternative  additive  form  would  take  each  factor  as  a  term  with  an  ap¬ 
propriate  coefficient.  In  this  case,  the  effectof  any  factor  with  a  very  low 
rating  would  be  absorbed  or  hidden  by  other  lactors  with  higher  ratings. 
This  is  minimized  in  the  form  adopted.  No  claim  is  made  that  a  single 
numerical  value  of  the  DEI  will  indicate  the  absolute  merit  of  a  particular 
display.  Rather,  the  DEI  values  of  alternative  displays  will  rank  the  de¬ 
signs  along  a  continuum  representing  information  transfer  effectiveness. 


Use  of  the  DEI  Technique 

In  applying  the  DEI  technique,  a  transfer  chart,  representing  the 
information  transfers  involved  in  a  given  display  reading  — 5>  information 
processing  — control  action  set,  is  first  drawn.  This  constitutes  a  major 
step  in  the  analysis  of  the  display  and  the  decisions  or  control  actions  to  be 
made. 


Transfer  of  information  between  the  display  and  the  decision  is  in¬ 
dicated  by  a  link  (line)  between  the  two.  If  intermediate  data  processing  is 
required,  this  is  represented  by  a  box  inserted  in  the  link  (creating  a  total 
of  two  links).  If  information  from  two  or  more  information  sources  is  re¬ 
quired  for  making  a  decision,  an  "and"  gate  symbol,  [•>  ,  is  used  to  con¬ 
nect  the  information  source  and  the  specific  decision.  If  a  decision  is  made 
on  the  basis  of  one  out  of  several  indicators  of  information,  an  "or"  sym¬ 
bol,  j\£>  ,  is  used  in  the  link  between  the  indicator  and  the  decision 

The  transfer  chart  is  not  a  flow  chart;  it  is  more  like  a  time  ex¬ 
posure  in  which  repeated  actions  do  not  produce  additional  lines  or  links. 

The  transfer  chart  serves  as  the  basis  for  calculating  each  of  the  factors 
used  in  the  DEI.  A  typical  transfer  chart,  drawn  in  this  str  is  presented 
in  Appendix  F  to  this  report. 


Factors  in  the  DEI 

The  complexity  factor,  involving  principles  1  and  4  above,  relates 
to  the  complexity  of  the  information  transfers  involved.  Each  link  is  assign¬ 
ed  a  weight  according  to  its  complexity.  A  box  possesses  a  link  weight  of 
four  while  other  link  weights  vary  between  zero  and  two.  The  rationale  for 
the  derivation  of  the  link  weights  in  accordance  with  the  probability  of  suc¬ 
cessful  information  transfer  has  been  presented  elsewhere  (Siegel,  Miehle, 
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and  Federman,  19fi2).  Since  the  sum  of  the  link  weights  (Zw)  may  equal 
zero,  this  factor  is  represented  as  l/l  +  Zw.  Its  greatest  value  is  then 
one,  and  it  approaches  zero  as  Zw  increases  indefinitely. 

The  directness  factor,  principle  2,  rests  upon  the  assumption  that 
in  an  "ideal”  system  only  one  link  will  connect  an  indicator  and  a  decision. 
Although  redundancy  (indicated  by  the  presence  of  an  symbol  in  the 

transfer  diagram)  reduces  the  probability  of  an  operator  error,  a  multi¬ 
plicity  of  links  to  or  from  a  source  of  information  is,  nevertheless,  an 
undesirable  feature.  The  factor  which  has  a  value  of  1  under  these  condi¬ 
tions  is 

(n  +  m) 

_ u 

2  N  (n  +  m)^ 

where  (n  +  m)u  is  the  sum  of  used  indicators  and  decisions,  (n  +  m)t  is  the 
total  number  of  indicators  and  decisions,  and  N  is  the  total  number  of  in¬ 
formation  links.  The  lower  limit  of  this  factor  is  zero. 

The  minimum  size  system,  possesses  one  indicator,  one  decision, 
and  no  logical  operator  functions.  These  aspects  are  covered  in  the  data 
transfer  factor,  principles  5  and  6.  They  suggest  the  form,  2/Q  +  nD, 
which  yields  a  value  of  unity  when  these  conditions  apply.  Here,  Q  is  the 
total  number  of  display  elements  plus  decisions  and  nQ  is  the  number  of 
gate,  mixer,  or  box  symbols  on  the  transfer  chart. 

The  match  factor,  principle  3,  enters  when  there  is  more  or  less 
information  supplied  than  is  needed  for  the  required  act.  The  mismatch  is 
this  difference.  The  match  factor  uses  the  sum  of  the  absolute  information 
differences  in  logarithmic  units  and  is  given  by 

[exp  (  -  I  Z  |M|  )  J 

For  a  perfect  match  Z  |m|  =  O,  therefore  exp  (O)  =  1  and  there  is  no  effect 
on  the  DEI. 


Exponential  Weighting  of  the  Factors 

The  weights  given  the  various  factors  were  first  determined  so  as  to 
give  maximum  agre  ment  vnih  the  judgment  of  the  developers  of  the  techni¬ 
que.  At  a  later  time,  the  DEI  was  enhanced  and  the  weights  determined  on 
the  basis  of  the  agreement  between  DEI  applications  to  a  number  of  systems 
with  the  opinions  of  tour  experts  who  evaluated  the  same  systems.  Since 
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the  DEI  is  composed  of  a  set  of  factors,  the  weights  are  the  exponents;  and 
since  the  values  resulting  are  usually  very  small,  for  interpretive  ease, 
fractional  exponents  are  used  to  raise  the  numerical  value  of  the  DEI  closer 
to  1. 

The  formula  of  the  DEI  is; 

(n  +  m)u  [exp  (  -  ~  E  |m|  )  ] 

(1  +  Z  )  y/  N  (n  +  M)  (Q  +  n  ) 
w  to 

This  can  be  calculated  by  hand  or  on  a  digital  computer  from  data  obtained 
directly  from  the  transfer  chart  and  the  task  application. 


Empirical  Valdity  of  the  DEI 

The  empirical  validity  of  the  DEI  technique  was  developed  by  ex¬ 
ponentially  weighting  the  factors  according  to  the  criteria  of  maximum 
agreement  between  the  mean  opinion  of  a  group  of  four  engineers  and  psy¬ 
chologists, and  the  result  of  the  DEI  technique  application  on  the  relative 
effectiveness,  from  the  information  transfer  point-of-view,  of  several  de¬ 
sign  variations  of  six  different  equipment  systems.  Later,  this  prelimi¬ 
nary  weighting  was  cross -validated  by  applying  the  technique  with  the  de¬ 
rived  weights  to  variations  of  six  other  equipments  and  comparing  the  ob¬ 
tained  DEI  values  with  the  composite  ranking  of  the  individual  equipments 
of  a  different  pool  of  four  human  factors  experts.  For  various  equipments, 
the  obtained  correlations  between  the  DEI  and  the  experts  were: 


Task  Correlation 


Radio  set  AN/GRC-50 

(radio  communications)  .  68 

Radio  set  AN/GRC-66 

(radio  relay  set  monitoring)  .82 

Radar  set  AN/ MPQ- 29  j 

(operating  adjustments)  .70  . 

Radar  set  AN/ MPQ- 29 
(target  acquisition  and 

radar  tracking)  ,70 
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Radar  set  AN/ TPS- 33 
(starting,  turning,  and 
orienting  equipment)  .  80 

Radar  set  AN/TPS-33 

(detection  of  targets)  .  60 


Discriminating  Power  of  the  DEI 

Discriminating  power  means  the  extent  to  which  a  metric  will  dis¬ 
tinguish  between  differences  in  a  variable.  In  the  current  application,  the 
interest  is  in  the  extent  to  which  the  DEI  distinguished  among  display  types 
or  presentation  techniques.  For  the  six  validations  described  above,  four 
or  five  equipment  variations  were  included  in  the  validational  sample  for 
each  of  the  equipments  considered.  Some  of  the  design  modifications  were 
quite  minor;  others  were  rather  extensive.  The  range  and  median  DEI 
values  for  each  equipment  were: 


Task 

Median 

Range 

>T 

o 

1 

to 

eg 

i 

o 

Radio  set  AN/GRC-50 

.0004 

.  0007 

Radio  set  AN/GRC-66 

.  525 

.  153 

Radar  set  AN/MPQ-29 
(operating  adjustments) 

.  067 

.020 

Radar  set  AN/MPQ-29 
(target  acquisition  and 
radar  tracking) 

.  311 

.  180 

Radar  set  AN/TPS-33 
(starting,  turning,  and 
orienting  equipment) 

.  103 

.  185 

Radar  set  AN/TPS-33 
(detection  of  target) 

.  152 

.  156 

These  data  strongly  suggest  that  the  technique  possesses  adequate 
sensitivity  for  distinguishing  between  variations  in  equipment  design. 
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Purpose  of  Present  Program 


The  prime  purpose  of  the  present  research  was  to  explore  the  use 
and  validity  of  the  DEI  technique  for  large  scale  command  and  control  dis¬ 
play  situations.  While  in  the  past  the  DEI  technique  was  used  to  evaluate 
displays  for  transferring  information  to  operators, and  for  the  operators  to 
act  on  this  information  (cf. ,  Siegel,  et  al. ,  1962,  1964),  in  the  present  re¬ 
search  the  DEI  technique  was  employed  to  evaluate  displays  for  transfer¬ 
ring  information  to  operators  and  for  the  operators  to  use  this  information 
for  decision-making  purposes.  By  decision  making,  we  mean  the  reception 
and  conversion  of  information  for  use  in  formulating  courses  of  action. 

To  this  end  the  present  study  investigated,  in  two  separate  experi¬ 
ments,  the  decisioning  ability  of  subjects  when  the  information  employed  in 
making  the  decision  is  presented  in  four  different  display  forms:  (1)  pic¬ 
torial  black-and-white,  (2)  pictorial-plus-color,  (3)  tabular  black-and-white, 
and  (4)  tabular-plus-color.  Employing  as  criterion  scores,  the  scores  of 
the  subjects  when  each  display  mode  was  employed,  the  DEI  values  for 
these  displays  were  compared  with  the  criterion  data.  The  resulting  re¬ 
lationships  were  considered  to  be  indicative  of  the  validity  of  the  DEI  for 
predicting  the  criterion. 

Previous  studies  have  pointed  to  the  value  of  color  in  visual  infor¬ 
mation  presentation  (Smith,  Farquhar,  and  Thomas,  1966;  Silver,  Landis, 
and  Jones,  1965).  The  results  of  these  investigators  suggested  that,  in 
visual  search  tasks,  items  presented  in  various  colors  are  more  easily 
discernible  than  others.  Hitt  (1961)  found  color  coding  to  be  superior  to 
such  other  coding  methods  as  alpha-numeric  classification,  geometric 
shape  coding,  and  configuration  for  several  different  tasks.  However,  most 
research  into  the  effectiveness  of  color  and  other  coding  methods  for  use  in 
large  scale  displays  has  involved  aspects  of  data  manipulation  that  are  some¬ 
what  less  complex  than  decision-making  and  dependent  variables  that  do  not 
involve  decision  correctness. 

The  present  research  investigated  the  effectiveness  of  color  in  large 
scale  command  and  control  displays  and  is  an  extension  of  previous  studies 
in  this  area, in  that  the  displays  considered  are  to  be  used  for  transferring 
information  needed  for  complex  decision-making. 

Smith  (1963)  found  that  error  scores  and  search  time  increased  with 
increasing  display  density.  Silver,  Landis,  and  Jones  (1965)  also  found  fact 
density  to  be  a  variable  having  critical  effects  upon  decision-making  ability. 

As  such,  it  was  decided  to  vary  fact  density  (numerosity  of  enemy  air  threats) 
across  three  levels  in  the  present  work. 
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Aside  from  the  effects  of  color  and  numerosity,  one  other  hypothesis 
was  formulated  and  investigated.  Two  different  types'  of  displays,  pictorial 
and  tabular,  were  involved.  It  was  hypothesized  that  each  is  of  value  for 
facilitating  different  levels  of  information  processing.  The  pictorial  display 
presents  a  complete,  holistic  representation  of  the  situation,  although  most 
pertinent  factors  are  In  a  symbolic  form.  This  display  was  hypothesized  as 
being  of  greater  value  for  complex  situational  judgments  where  the  integra¬ 
tion,  interpolation  or  extrapolation  of  information  from  several  sources  is 
'required.  On  the  other  hand,  the  tabular  display,  in  which  the  information 
is  presented  in  the  form  of  continuous  and  discrete  indicators  involving 
quantitative  values,  was  hypothesized  as  being  superior  for  determining 
simple  facts,  such  as  system  status,  availability  of  units,  and  level. 
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CHAPTER  II 


INITIAL  EXPERIMENT 


As  stated  in  Chapter  I,  the  primary  purpose  of  the  present  program 
was  to  validate  the  DEI  technique  for  predicting  the  ability  of  operators  to 
make  the  decisions  involved  in  the  command-control  type  of  situation  when 
large  scale  displays  were  employed.  Secondary  purposes  involved  compari¬ 
sons  of  the  power  of  two  display  types  (pictorial  and  tabular)  for  various 
types  of  information  processing  and  investigation  of  the  advantages,  if  any, 
of  color  coding  in  this  type  of  situation.  In  the  first  experiment,  four  dis¬ 
play  design  presentation  techniques  were  involved,  e.  g.  ,  pictorial  black-and- 
white,  pictorial-plus-color,  tabular  black-and-white,  and  tabular-plus-color. 
The  task  of  the  subjects  was  to  extract  information  from.and  make  decisions 
(through  the  employment  of  a  set  of  decision  rules)  on  the  basis  of  the  in¬ 
formation  presented  in  the  displays. 

An  ordered  series  of  six  stimulus  situations  was  developed.  Each 
pair  of  successive  stimulus  situations,  in  the  graduated  series,  was  con¬ 
sidered,  on  the  basis  of  the  number  of  threats  it  contained,  to  be  at  a  higher 
difficulty  level.  The  number  of  threats  contained  in  each  stimulus  was: 


Stimulus  Situation 
Number 

Difficulty  Level 

Number  of 
Threats 

1 

Easy 

4 

2 

Easy 

7 

3 

Intermediate 

10 

4 

Intermediate 

10 

5 

Difficult 

12 

6 

Difficult 

15 

For  each  stimulus  situation,  four  matched  displays  were  developed: 
(1)  pictorial  (Figure  1),  (2)  pictorial-plus-color,  (3)  tabular  (Figure  2),  and 
(4)  tabular-plus-color.  Each  display  conveyed  exactly  the  same  information 
for  a  given  stimulus  situation,  but  in  a  different  manner.  Since  six  stimulus 
situations  and  four  display  types  were  included,  a  total  of  24  stimuli  was  in¬ 
volved. 
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Stimuli 


The  stimulus  materials,  were  static  visual  displays,  representative 
of  large  scale  command  and  control  situations.  The  situations  involved  the 
air  defense  of  two  major  east  coast  complexes:  (1)  Philadelphia,  and  (2)  a 
Washington-Baltimore  composite. 


Pictorial  Display 

The  pictorial  displays  (Figure  1)  presented  both  qualitative  and 
quantitative  information  about  the  command  and  control  situation.  A  map 
of  the  two  major  east  coast  complexes  was  presented  in  the  sub-display 
lahelled"Threat  Status."  In  this  sub-display,  not  only  could  the  geography 
and  relative  positions  of  the  defending  batteries  and  attacking  aircraft  be 
readily  determined,  but  weather  information, as  well  as  other  threat  rele¬ 
vant  information.was  presented. 


Threat  Status 

Each  threat  was  identified  by  a  threat  number.  The  threat  number 
in  Figure  3  is  05.  Since  15  threats  are  the  maximum  number  that  could  ap¬ 
pear  in  any  display  (an  arbitrary  cut  point),  threat  tag  numbers  could  range 
from  01  to  15,  If  the  threat  number  was  prefixed  with  the  numeral  "5" 

(e.  g. ,  54),  the  additional  information  that  the  threat  contained  a  nuclear  war¬ 
head  was  provided.  The  color  displays  presented  this  information  redun¬ 
dantly.  For  these  displays, if  the  threat  was  believed  to  be  carrying  a  nuc- 
clear  warhead,  all  threat  information  was  also  presented  in  purple. 

If  a  battery  was  assigned  to  intercept  a  threat,  a  battery  assignment 
arc  appeared  above  the  threat  number.  If  no  battery  was  assigned  to  the 
threat,  the  battery  assignment  arc  was  omitted  from  the  display.  Following 
the  battery  assignment  arc  was  the  indication  of  the  altitude  of  the  threat. 

One  of  four  different  altitude  ranges  could  be  indicated.  One  dot  indicated 
an  altitude  of  0-14,  999  feet;  two  dots,  an  altitude  of  15,  000-29,  999  feet; 
three  dots,  the  range  of  30,  000-49,  999  feet;  and  four  dots  represented 
50,  000  feet  and  over. 

If  a  battery  assigned  to  a  threat  had  launched  a  missile  against  it, 
one  of  three  missile-launched  indicators  could  appear.  These  corresponded 
to  the  three  types  of  missiles  that  could  be  stored  in  the  battery's  stockpile. 

A  single  arc  (as  in  Figure  3)  represented  the  launching  of  a  low  explosive 
missile;  two  arcs  indicated  a  high  explosive  missile  launch;  and  three  arcs 
indicated  the  launching  of  nuclear  weapons.  Color  was  also  used  when 
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presenting  this  information  in  the  pictorial-color  displays.  Blue  was  used 
with  single  arcs,  red  with  double  arcs,  and  purple  with  triple  arcs. 

The  threat's  present  position  and  sizewerecommumcated  via  circles 
which  appeared  immediately  to  the  left  of  the  threat  number.  The  position 
of  this  circle  indicated  the  present  position  of  the  threat  and  allowed  extra¬ 
polative  estimations  of  the  distance  of  a  threat  from  the  center  of  a  defense 
complex.  The  extrapolation  of  distance  was  facilitated  through  the  use  of 
standard  marker  circles,  called  the  missile  range  circles.  These  repre¬ 
sented  a  distance  of  150  miles  from  the  center  of  a  complex.  In  the  pic¬ 
torial-plus-color  displays,  the  missile  range  circle  was  red. 

The  number  of  targets  or  hostile  aircraft  contained  in  any  threat 
could  vary  from  one  to  10  or  more.  If  a  threat  was  composed  of  a  single 
target,  a  solid  circle  appeared  (  0  ).  If  the  threat  contained  a  few  targets, 
that  is  from  two  to  nine  targets,  an  open  circle  (  ©  )  appeared.  If  there 
were  many  targets  (more  than  nine)  in  the  threat,  an  open  circle  with  a  band 
encircling  it  (  ©  )  was  presented.  The  band  was  colored  red  in  the  pictorial 
plus  color  displays. 

A  solid,  heavy  vector,  emanatingfrom  the  threats  "present  position" 
circle  locator,  provided  information  regarding  the  threat's  predicted  posi¬ 
tion,  after  15  seconds.  Thus,  the  speed  of  the  threat  could  be  estimated,  if 
necessary.  In  the  color-coded  version,  if  a  threat  was  inside  the  missile 
range  or  would  be  inside  within  15  seconds  and  still  remain  unassigned,  the 
predicted  range  bar  was  colored  brown. 

Leading  the  predicted  range  vector  was  the  directional  line.  This 
line  indicated  the  relative  direction  in  which  the  threat  was  heading  and  pro¬ 
vided  continuity  to  the  predicted  intercept  point  indicator,  a  diamond  head. 
The  predicted  intercept  point  indicator,  too,  was  colored  brown  if  the  threat 
was  unassigned  and  inside  the  missile  range, or  was  predicted  to  be  inside  of 
missile  range  within  15  seconds. 

The  final  unit  of  information  transferred  through  threat-related  codes 
was  the  designation  of  the  battery  assigned  to  intercept  a  threat.  A  letter  in 
front  of  an  intercept  point  indicated  the  battery  assigned  to  that  threat.  The 
actual  locations  of  the  batteries  were  communicated  by  alphabetic  indicators 
around  the  defense  complexes.  Batteries  A  through  H  were  the  eight  bat¬ 
teries  surrounding  the  Philadelphia  complex;  batteries  I  through  P  were  the 
eight  batteries  surrounding  the  Washington- Baltimore  complex.  A  battery's 
position,  around  a  complex,  was  never  changed.  In  addition  to  geographic 
location,  each  complex  was  identifiable  through  a  symbol,  readily  associated 
with  its  major  city.  A  replica  of  William  Penn  appeared  in  the  center  of  the 
Philadelphia  complex  and  a  replica  of  the  Washington  Monument  appeared  in 
the  center  of  the  Washington-Baltimore  complex. 
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Weather  over  the  area  was  also  indicated.  The  weather  symbots 
possessed  no  prescribed  position;  they  could  appear  any  place  in  the  display. 
An  open  circle,  Q  ,  indicated  clear  weather,  (5[)  indicated  rain,  (0  indicat¬ 
ed  snow.  Wind  direction  and  velocity  were  indicated  in  the  following  manner; 
a  line  cutting  the  perimeter  of  the  circle  indicated  the  direction  in  which  the 
wind  was  heading.  Barbs  at  the  end  of  this  line  indicated  wind  speed  in  miles 
per  hour.  Each  full  barb  represented  ten  miles  per  hour;  a  half  barb  repre¬ 
sented  five  miles  per  hour. 


Battery  Status 

Directly  below  the  threat  status  sub-display  were  battery  status  sub¬ 
displays,  one  for  each  complex.  All  the  information  specific  to  the  batteries 
protecting  the  complexes  were  contained  in  these  displays.  Each  battery's 
ready  status,  assignment  status,  type  of  missle  launched,  probability  of  kill¬ 
ing  the  threat,  weapons  available,  and  a  history  of  past  performance  were 
contained  in  the  battery  status  sub  display. 

If  a  battery  was  ready  to  receive  an  assignment,  an  arc  appeared 
above  and  to  the  left  of  the  battery  letter,  as  it  was  displayed  in  its  position 
around  the  complex.  In  the  color-coded  version, the  arc  and  the  letter  were 
green.  If  a  battery  was  assigned  to  a  threat,  an  arc  appeared  above  and  to 
the  right  of  the  battery  letter. 

Whenever  a  battery  had  launched  a  missile  against  a  threat,  the  mis¬ 
sile  launch  indicator  appeared  immediately  below  the  battery  letter.  These 
were  the  same  as  those  appearing  in  the  threat  status  sub-displays.  Again, 
color  was  used  where  indicated,  blue  for  low  explosive  missiles,  red  for  high 
explosive  missiles,  and  purple  for  nuclear  missiles. 

Information  was  provided  relating  to  the  kill  probability  for  particular 
threats,  i.  e.  ,  the  chances  a  battery  had  in  downing  a  particular  threat. 

These  codes  were  found  surrounding  the  battery  letter  in  its  position  around 
the  complex.  One  of  four  different  probability  estimates  could  appear  o= 
0.25,  Q  =0.  50,  -Q  =0.75,  0  =1.00. 

Alongside  the  geographic  complex  portions  of  the  battery  status  sub¬ 
displays,  graphic  exhibits  were  presented  containing  information  relative  to 
the  available  weapons  (missiles)  in  the  various  batteries  and  the  performance 
record  of  each  battery.  Bar  graphs  were  used  for  these  purposes.  For  the 
pictorial-plus-color  displays,  the  bars  were  color  coded  to  coincide  with  the 
three  types  of  missiles  stored  by  the  batteries.  The  graphic  displays  of  bat¬ 
tery  results  presented  a  history  of  each  battery's  performance  in  terms  of 
the  number  of  hits,  misses,  and  aborts  during  the  present  battle. 
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Computer  Status 

Two  computers  were  associatea  with  each  major  complex.  Three 
different  readiness  states  were  delineated  for  each  computer:  "ok,  " 
"standby,  "  and  "down.  "  Indicator  bars  pointed  to  the  legend  designating 
the  current  state  of  the  computer.  In  the  color  displays  a  green  bar  meant 
"ok,  "  a  red  bar  stood  for  "down,  "  and  a  black  bar  was  used  for  "standby.  " 
The  symbols  used  to  identify  Philadelphia  and  Washington-Baltimore  were 
the  same:  William  Penn  and  the  Washington  Monument. 


Missile  Types  Launched 

The  final  sub-displays  in  the  pictorial  presentations  referred  to  the 
missile  types  launched  by  the  batteries  in  each  complex.  These  were 
graphic  presentations  of  the  number  of  missiles  of  each  of  the  three  types, 
that  had  been  launched.  For  the  displays  involving  color  coding,  the  same 
colors  were  used  for  each  missile  type  as  had  been  used  in  the  previously 
discussed  sub-displays. 


Tabular  Displays 

The  counterpart  of  pictorial  display,  the  tabular  display  (Figure  2), 
directly  presented  all  the  information  contained  in  the  pictorial  displays. 
Here,  digital-counter,  alpha-numeric,  and  lensed- legend  billboard  and 
pointer  type  visual  displays  were  involved.  Precisely  the  same  information 
was  presented  on  the  tabular  displays  as  on  the  pictorial  displays.  Hence, 
for  any  given  situation,  threat  status,  battery  status,  and  missile  informa¬ 
tion,  etc,  ,  were  identical  across  the  two  display  types.  The  tabular  display 
contained  no  figural  or  pictorial  presentations, but  relied  exclusively  on  sym¬ 
bolic  presentation  of  information. 


Threat  Status 


All  relevant  information  pertaining  to  individual  threats  was  con¬ 
tained  in  individual  threat  status  blocks  (Figure  2).  Here,  the  threats  were 
identified  by  threat  number,  ranging  from  01  to  15.  A  threat  tagged  with  a 
number  starting  with  the  numeral  "5"  contained  a  nuclear  warhead.  Color 
was  used  in  the  tabular-plus-color  displays  to  indicate  a  threat  containing  a 
nuclear  warhead.  Thus,  a  nuclear  threat  was  indicated  by  both  a  tag, which 
started  with  the  number  "5,  "  as  well  as  the  presentation  of  this  information 
in  blue  (as  was  the  case  for  the  pictorial-plus-color  displays).  The  size  of 
the  threat  followed  the  threat  number  on  the  top  line  of  the  threat  status 
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block.  "Single,  "  "few,  "  or  "many"  were  used.  In  the  color  displays,  when 
"many"  was  used,  the  word  "many"  appeared  in  red. 

Altitude,  actual  range,  and  predicted  range  were  presented  through 
moving  pointer  and  three  or  four  drum  counter  displays.  The  pointers  indi¬ 
cated  qualitative  values  by  pointing  to  a  vertical  index  and  were  constructed 
so  that  there  was  a  "window"  at  the  tip  of  the  pointer  in  which  actual  quanti¬ 
tative  values  appeared  in  digital  form.  It  was  also  possible  to  read  these 
indicators  qualitatively  on  the  basis  of  the  height  of  the  vertical  "thermo¬ 
meter"  contained  in  each.or  on  the  basis  of  the  level  of  the  counter.  For 
the  tabular-plus-color  displays,  if  a  threat  was  unassigned  and  inside  the 
missile  range  or  was  going  to  be  inside  within  the  next  15  seconds,  the  pre¬ 
dicted  range  information  was  presented  in  brown  (as  in  the  pictorial-plus- 
color  displays). 

l 

Below  these  indicators,  battery  assignment,  missile  launch,  and  de¬ 
fense  complex  information  was  provided.  If  a  battery  had  not  yet  been  as¬ 
signed  to  a  threat,  the  complex  from  which  a  battery  was  to  be  selected  ap¬ 
peared  in  the  window  provided  under  "Complex.  "  This  would  be  either 
"Phila"  or  "Wash"  and  was  based  on  the  proximity  of  the  threat  to  each  com¬ 
plex.  An  assignment  was  made  to  a  battery  in  the  complex  closest  to  the 
threat.  If  a  battery  was  assigned  to  the  threat,  its  letter  appeared  in  the 
"Battery  Assigned"  box  and  the  "Complex"  box  remained  empty.  Since  the 
letters  A  through  H  were  designators  for  Philadelphia, and  I  through  P  for 
Washington- Baltimore,  further  identification  of  the  complex  was  not  pro¬ 
vided.  When  a  battery  launched  a  missile  against  a  threat,  the  type  of  mis¬ 
sile  used  appeared  in  the  "Missile  Launched"  box;  "L"  for  low  explosive, 

"H"  for  high  explosive,  and  "N"  for  nuclear.  In  the  color  displays  the  "L" 

was  blue,  the  "H"  was  red,  and  the  "N"  was  purple.  1 

The  actual  heading  and  the  coordinates  of  the  threat  were  presented 
in  a  digital  manner.  The  actual  heading  was  indicated  by  the  polar  heading 
of  the  threat  in  degrees,  as  measured  from  its  actual  or  present  position. 

The  actual  coordinates  indicated  the  present  position  of  the  threat  in  terms 
of  compass  bearings,  i.  e.,  N,  NE,  E,  SE,  S,  SW,  W,  and  NW.  Thus, suf¬ 
ficient  information  was  provided  to  locate  a  specific  threat.  The  same  in¬ 
formation  was  provided  for  the  predicted  heading  and  coordinates.  These 
were  heading  and  coordinate  predictions  for  15  seconds  hence. 

The  intercept  point  was  located  in  terms  of  range  and  coordinates.  • 

Range  was  indicated  in  miles  from  the  center  of  a  complex  and  the  coordi¬ 
nates  were  presented  in  compass  bearings. 

The  weather  indicators  were  specific  to  each  threat  Wind  informa¬ 
tion  was  given  by  a  counter-pointer  combination.  Wind  direction  was 
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indicated  by  the  pointer;  wind  velocity  was  presented,  in  miles  per  hour, 
by  the  digital  counter.  The  weather  conditions  appeared  in  a  lensed- legend- 
ed  indicator  immediately  below  the  wind  information.  One  of  three  weather 
conditions  could  appear:  rain,  snow,  or  clear. 


Battery  Status 

The  first  column  in  the  battery  status  billboard,  labelled  "Coor'd,  " 
provided  the  compass  bearings  of  the  batteries  around  the  complexes  (e.  g.  , 
N,  NE,  E,  SE,  S,  SW,  W,  and  NW).  The  battery  designators  appeared  in 
the  second  column.  In  the  tabular-plus-color  displays,  when  a  battery  was 
ready  to  receive  an  assignment,  its  letter  in  the  second  column  appeared  in 
greet*.  In  addition,  all  ready  batteries  were  indicated  by  a  solid  box  in  the 
third  column.  This  indication  was  green  for  the  tabular-plus-color  displays. 
Those  batteries  already  assigned  were  designated  with  a  "Y"  (yes)  in  the 
fourth  column;  unassigned  batteries  had  a  "N"  (no)  in  the  assigned  column. 

Missiles  launched  by  batteries  were  indicated  in  the  next  column. 

"L"  was  used  for  low  explosive,  "H"  for  high  explosive,  and  "N"  for  nuclear. 
The  tabular-plus-color  displays  presented  this  information  in  blue,  red,  and 
purple,  respectively.  The  kill  probability  was  presented  in  the  following 
column.  The  probability  appeared  alongside  the  battery  assigned  to  the 
threat  for  which  the  kill  probability  had  been  calculated. 

Digital  counter  displays  were  used  to  provide  the  weapons  available 
and  battery  results  information.  Color  coding  was  used  in  displaying  the 
weapons  available  in  the  color  displays.  These  were  again  blue,  red,  and 
purple  for  low  explosive,  high  explosive,  and  nuclear  weapons.  Battery 
results  received  no  color  coding. 


Computer  Status 

Computer  status  was  provided  through  one  of  three  simulated  lensed- 
legend  indicators.  These  could  be  read  "ok,  "  "standby,  "  or  "down.  "  Green 
was  used  for  "ok,  "  black  was  used  for  "standby,  "  and  red  was  used  for 
"down"  in  the  color  displays. 


Missile  Types  Launched 

The  total  number  of  each  type  of  missile  launched  by  a  complex  was 
provided  in  the  missile  types  launched  sub-display.  Blue,  red,  and  purple 
were  used  to  identify  the  appropriate  missile  type  in  the  tabular-plus-color 
displays. 
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Apparatus 


The  displays  of  the  command  and  control  situations  were  prepared 
in  lantern  slide  form.  The  use  of  lantern  slides  allowed  for  projected 
images  of  adequate  definition  and  a  total  picture  size  of  four  feet  by  five 
and  one-quarter  feet.  Thus,  the  subjects  were  able  to  read  the  displays, 
with  relative  ease,  at  viewing  distances  of  from  six  to  ten  feet.  The  colors, 
in  both  the  pictorial  and  tabular  displays,  were  adequately  bright  and  satu¬ 
rated  against  their  white  screen  background. 

The  subjects  were  seated  in  a  quiet,  semi-darkened  room  and  pro¬ 
vided  with  lap  boards  to  support  their  writing  materials.  The  arrangement 
of  seats  in  front  of  the  projection  screen  was  constant  for  all  groups  of  sub¬ 
jects.  To  minimize  the  displacement  and  distortion  of  the  projected  images, 
the  seats  were  arranged  so  that  none  was  at  an  acute  angle  to  the  screen. 


The  Decisions 


The  visual  displays  provided  the  information  for  a  series  of  decisions, 
which  the  subjects  were  required  to  make  on  the  basis  of  the  information  dis¬ 
played.  The  adequacy  of  these  decisions  served  as  an  indication  of  the  ability 
of  the  display  to  transfer  information.  Two  "decision  levels"  were  used. 

The  more  difficult  level,  referred  to  as  "decision-making,  "  was  complex 
enough  to  require  the  integration  of  several  different  aspects  of  the  infor¬ 
mation  provided.  The  other  level,  "information  availability,  "  was  less  com¬ 
plex  and  merely  required  the  subjects  to  read  or  extract  information  from 
the  displays.  Thirty  decision  questions  were  answered  by  each  subject  for 
each  display;  twenty  were  of  the  information-availability  type  and  ten  were 
of  the  mere  complex  decision-making  type.  Examples  of  questions  classified 
under  decision  level  I,  information  availability,  were:  How  many  batteries 
are  unassigned?,  How  many  threats  are  above  29,  999  feet?.  How  many 
threats  contain  nuclear  warheads?  Examples  of  decision-making,  decision- 
level  II,  questions  are:  Which  weapon  should  battery  A  launch  against  threat 
52?,  To  which  battery  should  threat  08  most  probably  be  assigned?,  If  both 
Philadelphia  computers  were  to  fail  now,  which  threat  would  you  assign  first 
to  the  Washington-Baltimore  complex?  A  complete  list  of  the  questions,  for 
each  stimulus,  appears  in  Appendix  A  to  this  report. 


Decision  Rules 


The  logic  for  making  the  decisions  was  provided  to  the  subjects  in 
the  form  of  a  set  of  decision  rules  or  ground  rules.  These  ground  rules  are 
presented  as  Appendix  C  to  this  report.  The  ground  rules  included 
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instructions  for  making  decisions  on  battery  assignment,  weapon  selection, 
wind  and  weather  considerations,  target  selection,  etc,  For  example,  in 
determining  if  a  nuclear  missile  should  be  launched  against  a  threat,  a  sub¬ 
ject  was  required  to  consider  and  see  that  the  following  conditions  were 
satisfied:  (1)  the  wind  was  not  in  the  direction  of  the  complex,  (2)  the  threat 
had  a  few  targets  (2  to  9)  or  many  targets  (10  or  more),  (3)  the  threat  con¬ 
tained  a  nuclear  warhead,  (4)  the  threat  was  above  14,  999  feet,  and  (5)  the 
predicted  intercept  point  was  or  would  be  more  than  50  miles  from  the  cen¬ 
ter  of  a  complex.  As  a  second  example,  in  determining  which  battery  to 
assign  to  a  particular  threat,  the  decision  rules  stated  that  a  threat  should 
be  assigned  to  the  nearest  unassigned  battery  with  at  least  five  of  the  appro¬ 
priate  weapon  type  available.  If  two  batteries  were  equidistant  from  the 
threat, and  each  had  at  least  five  of  the  appropriate  weapons  available,  the 
battery  was  to  be  selected  that  had  the  highest  "batting  average.  "  Batting  i  i 
average  was  determined  by  using  the  following  formula  :  1 


Batting  average  = 


Hits  (this  battle) _ 

Misses  +•  Aborts  (this  battle) 


The  training  of  the  subjects  in  the  use  and  application  of  the  ground 
rules  is  presented  in  a  later  section  of  the  present  chapter. 


Subjects 

Twenty  male  subjects  were  used  in  the  experiment.  They  were  all 
undergraduates  at  two  local  colleges,  Eastern  Baptist  and  La  Salle.  The 
subjects  were  assigned  randomly  to  one  of  four  groups.  However,  no  more 
than  three  subjects  from  either  college  were  assigned  to  any  one  group,  j 
thus  controlling  for  any  college  specific  bias  which  might  have  been  involved. 
A  group  was  composed  of  five  subjects.  Each  group  was  identified  with  a 
particular  display  presentation  technique  (pictorial,  pictorial-plus-color, 
tabular,  and  tabular-plus-color)  and  all  the  stimuli  presented  to  the  group 
were  of  its  respective  type.  All  subjects  had  normal  color  vision. 

The  subjects  were  paid  $1.  50  per  hour  for  their  participation. 


Subject  Training 

At  the  first  of  three  sessions,  conducted  separately  with  each  of  the 
four  groups,  the  subjects  were  given  a  brief  semi-structured  introductory 
lecture.  In  this  lecture,  the  purpose  of  the  study  and  the  experimental  pro¬ 
cedures  were  explained.  The  subjects  were  informed  that  Applied 
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Psychological  Services  was  under  contract  with  the  Air  Force  to  perform 
research  into  the  effectiveness  of  different  types  of  visual  displays  for 
transferring  information  and  the  subsequent  use  of  this  information  for 
making  decisions.  Since  air  defense-type  programs  were  selected  as 
vehicles  for  studying  the  problems  at  hand,  the  subjects  were  to  think  of 
themselves  as  assignment  officers. 

In  addition,  the  following  points  were  covered  in  the  initial  lecture. 
The  static  displays  were  to  be  thought  of  as  photographs.or  stills, of  dynamic 
and  constantly  changing  situation.-  in  a  air  defense  circumstance.  As  as¬ 
signment  officers,  the  subjects  L.  d  control  over  two  complexes- -Philadel¬ 
phia  and  Washington- Baltimore.  The  training  program  at  this  first  session 
involved  learning  to  read  the  displays  and  training  in  the  use  of  the  ground 
rules.  The  pay  was  $1. 50  per  hour  and  a  bonus  would  be  given  to  those  who 
achieved  a  criterion  score. 

The  task  of  the  subjects  was  ampliffed  and  they  were  told  that  there 
would  fce  six  independent  command  and  control  problem  situations.  They 
would  answer  questions  on  each  command  and  control  problem.  Before  the 
actual  training  session  began,  the  subjects  were  further  informed  that  the 
first  session  was  primarily  for  training;  a  second  session  for  both  re¬ 
training  (during  which  a  complete  review  of  all  the  material  covered  in  ses¬ 
sion  I  would  occur)  and  administration  of  one-balf  of  the  experimental  prob¬ 
lems  was  involved;  and  in  the  third  session  the  other  half  of  the  experi¬ 
mental  problems  would  be  administered. 

Since  learning  ability  war  not  an  experimental  variable,  it  was  con¬ 
sidered  necessary  to  allow  for  as  much  training  as  was  necessary  to  bring 
the  subjects  to  a  satisfactory  initial  level  of  proficiency. 

A  folder  containing  display  reading  instruction  sheets  and  the  de¬ 
cision  ground  rules  was  then  distributed.  The  display  reading  sheets  pre¬ 
sented  all  the  display  symbology  (and  color  coding  for  the  "color"  groups) 
to  be  learned. 

Examples  of  such  symbolism  found  in  the  pictorial  displays  were 
the  altitude  readings  (i.  e.  ,  •  =  0-14,  999  feet,  I  =  15,  000-29,  999  feet), 

the  missle  launched  arcs  (  s_/,  =  low  explosive  weapon,  ^  =  nuclear  wea¬ 
pon),  and  threat  size  (  #  =  single  target,  O  =  few  targets).  Examples 
of  codes  in  the  tabular  displays  were  misf-iles  launched  (L  =  low  explosive, 

N  =  nuclear),  actual  coordinates  (N  =  norih,  S  =  south),  and  battery  as¬ 
signment  status  (Y  =  yes,  N  =  no). 

Then,  an  especially  constructed  training  slide  was  projected  on  a 
screen  and  the  display  coding  and  ground  rules  were  covered  in  detail. 
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Questions  were  encouraged  and  answered.  The  subjects  were  then  told  to 
take  the  ground  rules  and  display-reading  instruction  sheets  with  them  and 
to  study  these  sheets  before  the  next  meeting.  They  were  advised  that  they 
would  be  tested  on  the  material  involved  and  that  they  would  be  paid  for  two 
hours  of  study  time. 

The  first  part  of  the  second  session  repeated  the  training  given  in 
session  I,  using  the  same  training  slide.  Then  a  test,  appearing  in  Ap¬ 
pendix  D  to  this  report, was  administered.  Only  a  few  errors  were  made  on 
tb<»  test.  The  frequency  distribution  of  test  scores  was: 

Scores  f 

96-100% 

91-95 
86-90 
81-85 


The  test  was  scored  immediately  and  errors  were  corrected  with  the  group. 
At  this  point  the  subjects  within  a  group  were  considered  ready  for  parti¬ 
cipation  in  the  first  data  collection  session. 


Display  Administration 

Immediately  following  the  second  training  session,  the  first  three  of 
the  six  research  stimulus  situations  were  administered.  The  second  three 
display  situations  were  administered  in  a  third  session  a  day  or  two  later. 
Each  session  lasted  two-three  hours,  including  five-minute  breaks  after 
each  hour. 


The  order  of  presentation  of  the  six  stimulus  situations  was  system¬ 
atically  counterbalanced  so  as  to  distribute  any  practice  or  fatigue  effects 
systematically  over  the  data.  Thus,  each  group  received  its  problems  in 
the  same  sequence  which  was  based  on  difficulty  level-- "easy,  "  "inter¬ 
mediate,  "  "difficult,  "  "difficult,  "  "intermediate,  "  "easy.  " 


The  procedure  for  exposing  the  stimuli  and  eliciting  answers  to  the 
decision  questions  was  based  on  the  results  of  a  pilot  study  (described  be¬ 
low).  After  distributing  the  question  booklets,  the  group  was  allowed  15 
seconds  to  read  the  first  question  to  be  answered.  The  first  slide  was  then 
projected  for  30  seconds  during  which  time  the  subjects  were  expected  to: 
extract  the  information  required  for  reaching  the  necessary  decision,  reach 
the  decision,  and  enter  the  decision  on  an  answer  sheet.  Thus,  for  any  one 
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question,  effectively,  45  seconds  were  allowed.  Following  the  answering 
of  the  first  question,  the  screen  was  blanked  and  the  subjects  read  the  next 
question  to  be  answered.  Then  the  slide  was  re-exposed  for  30  seconds. 
This  procedure  was  repeated  until  the  30  decisions  had  been  reached  for  a 
slide.  After  making  30  decisions  for  a  slide,  the  answer  sheets  were  col¬ 
lected,  new  answer  sheets  distributed,  and  the  procedure  was  repeated  for 
the  next  stimulus  situa' >on. 

Each  subject  answered  30  questions  for  each  slide, and  six  slides 
were  involved  for  each  display  type.  Thus,  180  decisions  were  obtained  per 
subject. 


Pilot  Study 

After  the  command  and  control  situations  were  constructed,  but 
prior  to  their  drafting  and  transposition  to  lantern  slide  form,  a  pilot  study 
was  performed.  Two  male  college  students  were  used  as  subjects  in  the 
pilot  study.  The  primary  purpose  of  the  pilot  study  was  to  verify  the  feasi¬ 
bility  of  the  research  administrative  procedures,  the  decision  questions, 
and  the  instructions. 

The  subjects  were  presented  with  several  of  the  command  and  con¬ 
trol  situations.  In  each  case  the  subjects  were  required  to  answer  the  thirty 
decision  questions.  One  subject  was  shown  only  the  pictorial-plus-color  dis¬ 
plays;  the  other  only  the  tabular-plus-color  displays.  The  pilot  study  served 
to  verify,  and, in  some  cases  modify,  the  study  procedures  to  be  used. 


Results  and  Discussion 

Each  subject's  decision  scores  (number  of  current  responses)  were 
calculated.  The  decision  scores  were  based  on  information  availability  (de¬ 
cision  level  I),  decision-making  (decision  level  II),  and  total  (the  sum  of  both 
decision  levels)  responses.  Separate  analyses  were  performed  on  these  three 
data  sets. 


Variance  Analyses 

Initially,  the  variables  of  concern  were  the  effect  of  display  type  (pic- 
tui  ial,  pictorial-plus-color,  tabular,  and  tabular- plus-color ),  and  difficulty 
level  among  the  six  command  and  control  situations.  Consequently,  these 
variables  were  subjected  to  a  variance  analysis.  Tables  1,  2,  and  3  present 
the  summaries  of  these  analyses  of  variance. 
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Table  1 

Summary  of  Analysis  of  Variancefor  Decision  Level  I  (Information  Availability) 


Source  of  Variation 

SS 

df_ 

MS 

_F 

Problems  (P) 

108.  94 

5 

21.  79 

7.  50* 

Display  Types  (DT) 

35.  36 

3 

11. 79 

3. 82** 

P  x  DT 

41.  89 

15 

2.79 

Error 

296.  80 

96 

3.09 

TOTAL 

482. 99 

119 

♦Significant  at  or  below  the  .  01  level  of  confidence 
♦♦Significant  at  or  below  the  .05  level  of  confidence 

Table  2 

Summary  of  Analysis  of  Variance  for  Decision  Level  II  (Decision  Making) 


Source  of  Variation 

SS 

_df 

MS 

_F 

Problems  (P) 

113. 

74 

5 

22.  75 

13.  23* 

Display  Types  (DT) 

56. 

96 

3 

18.  99 

11. 04* 

P  x  DT 

51. 

69 

15 

3.  45 

2.01** 

Error 

165. 

60 

96 

1.72 

TOTAL 

387. 

99 

119 

♦Significant  at  or  below  the  .01  level  of  confidence 
♦♦Significant  at  or  below  the  .05  level  of  confidence 

Table  3 


Summary  of  Analysis  of  Variance  for  Both  Decision  Levels 


Source  of  Variation 

SS 

df 

MS 

_F 

Problems  (P) 

314.  37 

5 

62.  37 

10.  17* 

Display  Types  (DT) 

72.  30 

3 

24.  10 

3.  90** 

P  x  DT 

73.  70 

25 

4.  91 

Error 

593.  60 

!  6 

6.  18 

TOTAL 

1053.  97 

1  i  9 

♦Significant  at  or  below  the  .01  level  of  confidence 
♦♦Significant  at  or  below  the  .05  level  of  confidence 
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The  analyses  of  variance  resulted  in  between  problem  F  ratios, 
which  were  significant  at,or  below  the  .  01  level  of  confidence  for  each  de 
cision  level  analyzed  individually,  and  for  the  total  across  both  levels. 


Difficulty  Level 

The  significant  differences  among  problems  support  the  finding  of 
Silver,  Landis,  and  Jones  (1965)  and  indicate  that  poorer  decisions  result 
as  threat  numerosity  (fact  density)  increases. 

Tukey's  test  for  significant  gaps  among  individual  means  was  ap¬ 
plied  to  each  of  the  data  sets.  For  decision  level  I,  the  results  indicated 
two  significantly  different  groups  (p  <.05):the  "easy"  and  "intermediate" 
problem  scores,  which  were  not  significantly  different  from  each  other, 
composed  the  higher  mean  information  availability  score  group,and  the 
lower  mean  information  availability  score  group  was  composed  of  the  "dif¬ 
ficult"  problems.  For  decision  level  II,  there  were  three  significant 
groups:  the  "easy"  problems  (highest  mean  decision  score  group)  were  sig¬ 
nificantly  different  from  the  "intermediate"  problems  (p<r.01),and  the  "dif¬ 
ficult"  problems  (p  ^.Ol),  and  the  "intermediate"  problems  were  signifi¬ 
cantly  different  from  the  "difficult"  problems  [  lowest  mean  decision  score 
group  (p  <" .  05)  ].  For  the  total  of  both  decision  levels,  again  there  were 
two  statistically  significant  groups;  the  "easy"  and  "intermediate"  problems 
although  differing  significantly  from  the  "difficult"  problems  (p  -r.01)  did 
not  differ  significantly  from  each  other. 

Figure  4,  presents  the  effect  of  increasing  difficulty  level  on  de¬ 
cision  score.  Threat  numerosity  seemed  to  have  had  a  greater  effect  on  de¬ 
cision  level  II  than  on  decision  level  I.  Lesser  effects  on  decision  score 
were  observed  between  the  "easy"  and  "intermediate"  levels  of  difficulty 
than  between  the  "intermediate"  and  "difficult"  levels. 


Display  Type 

The  analyses  of  variance  resulted  in  significant  between-display  type 
F  ratios  at  either  the  .  05  or  .  01  levels  of  confidence  for  both  levels  of  de¬ 
cision  making  and  for  the  total  across  both  levels.  The  Tukey  test  for  sig¬ 
nificant  gaps  among  the  treatment  means  indicated  two  significantly  different 
groups  (  p  <,01)  for  decision  level  I.  The  higher  mean  decision  score  grouo 
consisted  of  both  tabular  displays  and  the  pictorial-plus-color  display.  The 
lower  group  consisted  of  the  pictorial  black-and-white  display  type.  In  de¬ 
cision  level  II,  two  significantly  different  groups  (p  <.01)  were  also  identi¬ 
fied.  Pictorial-plus-color  displays  were  significantly  different  from  the 
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other  three  display  types.  When  the  total  for  both  decision  levels  was  con¬ 
sidered,  the  pictorial-plus-color  display,  with  the  highest  mean  decision 
score,  was  significantly  different  from  the  tabular-plus-color  and  the  pic¬ 
torial  black-and-white  displays.  Figure  5  presents  the  mean  decision 
scores  for  the  two  decision  levels  and  the  mean  total  score,  for  each  dis¬ 
play  type. 

The  scores  on  the  pictorial  and  the  tabular  displays,  at  the  various 
difficulty  levels,  are  presented  in  Figure  6.  These  figures  suggest  that 
for  decision  level  I,  information  availability,  the  tabular  displays  were 
superior  to  the  pictorial  displays  at  all  levels.  This  finding  is  believed 
to  reflect  the  nature  of  the  information  availability  questions  which  involved 
direct  read-outs  from  the  displays  without  comparisons,  extrapolations,  in¬ 
terpolations,  and  the  like.  However,  where  the  decisions  were  more  com¬ 
plex  and  involved, the  integration  of  several  components  of  the  display, 
rather  than  just  reading  out  information,  the  pictorial  display  seemed  to 
have  an  edge  in  the  direction  of  improving  decision-making.  From  the 
graph  of  Figure  6, which  shows  the  relationship  across  both  decision  levels, 
the  pictorial  display  seems  to  be  generally  superior.  Thus,  the  gross  state¬ 
ment  that  pictorial  displays  generally  facilitate  complex  decision-making 
seems  supported. 


Color  Coding 

The  effect  of  color  on  each  of  the  display  types  is  indicated  in  Fig¬ 
ure  5, and  the  effect  of  color  on  each  difficulty  level  within  each  decision 
level  is  shown  in  Figure  7.  Figure  5  suggests  that  color  had  a  positive  ef¬ 
fect  when  the  pictorial  type  display  was  employed.  Color  did  not  seem  to 
influence  decision  scores  positively  when  the  tabular  displays  were  in¬ 
volved.  Figure  7  suggests  that  color  helped  decision-making  more  than  in¬ 
formation  extraction.  An  increase  of  19  per  cent  was  attained  in  the  de¬ 
cision-making  score  when  color  was  used  in  the  pictorial  display.  There 
was  a  five  percent  increase  in  the  score  when  information  extraction  was 
involved.  The  effect  of  color  seems  to  hold  quite  generally,  but  it  is  noted 
that  it  did  not  seem  to  hold  for  the  easy  problems  of  decision  level  I.  It  is 
quite  possible  that  information  extraction,  at  the  simple  level  and  when  ade¬ 
quate  time  is  allowed,  is  easy  enough  as  to  be  unaffected  by  color  coding. 


Redundancy 

The  effect  of  redundancy  of  information  on  the  information  avail¬ 
ability  items  (decision  level  I)  was  examined  further.  Figure  8  indicates 
that,  in  general,  as  redundancy  increased  from  zero  to  three  (the  maximum 


27 


DIFFICULTY  LEVEL 


FIGURE  7.  DECISION  SCORES  FOR  COLOR  AND  BLA 
WHITE  DISPLAYS. 
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amount  of  redundant  information),  there  was  an  increase  in  the  number  of 
correct  responses,  Redundancy  of  information  was  determined  by  the  num¬ 
ber  of  symbols  and  sources  providing  the  same  information.  For  example, 
in  the  pictorial  display,  a  ready  battery  was  indicated  by  a  ready  arc;  if 
the  display  was  in  color,  the  ready  arc  and  the  battery  letter  were  presented 
in  green.  Thus,  in  the  pictorial  black-and-white  display  type,  the  decision 
pertaining  to  the  number  of  ready  batteries  was  considered  as  having  zero 
redundancy.  However,  in  the  pictorial  color  display,  redundancy  of  ready 
status  information  was  considered  as  having  a  value  of  two.  In  general,  the 
anticipated  positive  effect  of  redundancy  on  information  extraction  is  indi¬ 
cated  by  Figure  8. 

Application  of  the  DEI  Technique 

The  Display  Evaluative  Index  technique  (DEI)  was  applied  to  the  six 
command  and  control  problems,  and  to  each  of  the  four  display  types  through 
which  the  information  was  conveyed.  The  DEI  results  in  a  hierarchical 
ordering  of  displays.  The  resulting  values  do  not  relate  displays  on  an  abso¬ 
lute  scale,  but  rather  on  a  relative  scale, or  continuum, purporting  to  repre¬ 
sent  the  information  transfer  effectiveness  for  the  purpose  of  decision-mak¬ 
ing.  Table  4  presents  the  indexes  resulting  from  application  of  the  DEI 
technique  to  the  display  types.  An  example  of  the  computations  of  a  DJEI  is 
presented  in  Appendix  F.  The  resulting  DEI  values  are  presented  in  rank 
order,  in  both  columns  and  rows,  from  high  to  low.  The  fact  that  the  DEI 
resulted  in  a  higher  value  for  the  less  difficult  problems  reflects  the  effect 
of  numerosity  on  information  transfer  effectiveness.  This  tendency  is  sup¬ 
ported  by  the  data  presented  earlier. 

Table  4 


DEI  Values  for  Problems 

and  Display  Typ 

es 

Pictorial- 

Pictorial 

Tabular- 

T  abular 

Problem 

plus-Color 

Black-and -White 

plus-Color 

Black-and-Whit 

1 

.  276* 

.  208 

.  202 

.  147 

2 

.  267 

.  195 

.  145 

.  107 

3 

.  200 

.  144 

.  102 

.  078 

4 

.  151 

.117 

.  099 

.  076 

5 

.  119 

.  093 

.  091 

.  071 

6 

.  106 

.  082 

,  040 

.  031 

* 

For  the  calculation  of  this  DEI  value,  see  Appendix  F.  All  other  DEI 
values  were  similarly  calculated. 


In  addition  to  ranking  the  displays  in  an  order  of  merit  relative  to 
problem  difficulty  level,  the  DEI  also  ranked  the  display  types  within  each 
problem.  Thus,  pictorial-plus-color,  pictorial  black-and-white,  tabular- 
plus-color,  and  tabular  black-and-white  was  the  relative  rank  order  ob¬ 
tained  from  the  application  of  the  DEI. 

Color, as  opposed  to  black  and  white, resulted  in  higher  DEI  values, 
because  of  its  effect  as  an  enabling  factor.  This  tendency  was  also  seen  in 
the  objective  decision  scores  reported  earlier.  That  is,  whenever  color 
provided  relevant, rather  than  redundant, information,  it  enabled  the  decision¬ 
making  process  and  also  reduced  the  number  of  display  elements  needed  to 
reach  a  specific  decision. 

The  pictorial  displays  showed  greater  DEI  values  than  the  tabular 
displays.  This  tendency  was  also  seen  in  the  more  objective  decision  data 
reported  earlier.  The  factors  in  the  DEI  causing  the  pictorial  displays  to 
rank  above  the  tabular  displays  were  the  Q  factor  and  the  mismatch  factor. 
The  DEI  penalizes  displays  which,  although  providing  the  same  information 
needed  to  make  the  same  decisions,  use  a  larger  number  of  display  elements 
to  do  it.  Similarly,  the  DEI  will  penalize  displays  which  provide  more  in¬ 
formation  than  is  needed  to  reach  certain  types  of  decisions,  as  was  the 
case  in  decision  level  I.  In  this  experiment,  the  information  presented  via 
the  tabular  displays  used  more  display  elements  (more  noise)  than  were 
needed  for  making  specific  decisions. 


Cori  elational  Analyses 

For  purposes  of  relating  the  obtained  DEI  values  and  the  empirical 
resubs  of  the  experiment,  as  measured  by  the  decision  scores,  a  series  of 
correlations  was  performed.  Correlational  analyses  were  performed  on 
the  data  representing  decision  level  I  by  display  type,  decision  level  II  by 
display  type,  the  total  for  both  decision  levels  by  display  type,  decision 
level  I  across  the  four  display  types,  decision  level  II  across  display  types, 
and  the  total  for  both  decision  levels  across  display  types. 

Triserial  correlations  were  used  *o  obtain  these  measures  of  re¬ 
lationship  (Jaspen,  1946).  The  triserial  correlation  is  a  useful  statistic 
when  one  variable  is  continuously  distributed  (decision  score)  and  the  other 
is  segmented  into  thr  ee  categories  (difficulty  level).  Criterion  ratings  of 
"easy,  "  '  intermediate,  "  and  "difficult"  problems,  which  were  based  on  the 
DEI  calculations  of  the  command  and  control  displays,  were  correlated 
against  the  appropriate  decision  score.  The  obtained  triserial  correlations 
are  presented  in  Tables  5,  6,  7,  and  0. 
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Table  5 


Triserial  Correlation  Coefficients  Between  Criterion 
Ratings  and  Decision  Scores  for  Decision  Level  I 

Display  Type 

Correlation 

Pictorial  plus  color 

.27 

Pictorial  black  and  white 

.  17 

Tabular  plus  color 

10 

Tabular  black  and  white 

.  22 

Table  6 

Triserial  Correlation  Coeffici 

ents  Between  Criterion 

Ratings  and  Decision  Scores 

for  Decision  Level  II 

Display  Type 

Correlation 

Pictorial-  plus-color 

.  66 

Pictorial  black  and  white 

.  68 

Tabular  plus  color 

.  42 

Tabular  black  and  white 

.  32 

Table  7 

Triserial  Correlation  Coefficients 

Between  Criterion  Ratings 

and  Decision  Scores  for  the  Total 

of  Both  Decision  Levels 

Display  Type 

Correlation 

Pictorial  plus  color 

.  28 

Pictorial  black  and  white 

.  29 

Tabular  plus  color  . 

.  08 

Tabular  black  and  white 

.  17 

Table  8 

Triserial  Correlation  Coefficients 

Between  Criterion  Ratings 

and  Decision  Scores  Across  Display  Types 

Decision  Level 

Correlation 

Decision  level  I 
Decision  level  II 
Total 
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.  13 
.  45 
.  20 


It  becomes  apparent  that  the  magnitude  of  the  correlation  coeffi¬ 
cients,  although  not  exceedingly  high,  support  a  contention  favoring  the  pre¬ 
dictive  validity  of  the  DEI  technique  as  an  objective  and  impartial  calcula- 
tional  device  for  quantitatively  determining  the  ability  of  displays  to  trans¬ 
fer  information  to  an  operator  and  for  the  operator  to  use  this  infot  mation 
for  decision-making.  The  correlation  coefficients  reveal  a  relatively  close 
relationship  between  the  DEI  and  decision  level  II  scores,  as  indicated  in 
Tables  6  and  8.  Also  from  Tables  5  through  7,  it's  apparent  that  the  DEI 
correlated  more  closely  with  the  decision  scores  resulting  from  the  pic¬ 
torial  displays  rather  than  the  tabular.  The  lowered  correlations  between 
the  DEI  values,  and  the  information  availability  scores  are  believed  to  be  an 
artifact  of  the  content  of  the  DEI.  The  DEI  is  based  on  considerations  which 
are  more  involved  in  decision-making  and  not  on  information  extraction  con¬ 
siderations. 


Probability  or  Information  Transfer  Analysis 

A  display  may  be  viewed  as  a  device  for  conveying  information  from 
a  source  to  an  operator.  The  quality  of  the  display,  or  its  effectiveness, can 
be  determined  from  its  capacity  to  convey  the  information  correctly.  The 
concepts  of  information  theory,  which  are  derived  essentially  from  Bayes' 
theorem.but  use  it  in  a  logarithmic  form,  can  be  used  to  measure  the  effec¬ 
tiveness  of  various  displays. 

Consider  first  the  weapon  assignment  problems.  These  were  solved 
through  assignment  of  a  low  explosive  (L),  or  high  explosive  (H),  or  nuclear 
(N)  warhead.  We  consider^  to  be  the  input  information  presented  on  the 
display  (x  indicates,  L,  H,  or  N)  and  ^  is  the  assignment  made  by  the  opera¬ 
tor  is  also  either  L,  H,  or  N).  Data  are  collected  with  a  display  using 
different  problems  and  different  operators.  From  these,  the  conditional 
probabilities  px  (y)  are  determined.  Thus  we  determine  p^  (L),  p^  (H), 
and  pj_  (N),  the  probability  that  when  a  low  explosive  warhead  should  be  as¬ 
signed,  the  operator  actually  assigns  L,  H,  and  N,  respectively  The  other 
conditional  probabilities  are  also  determined  from  the  experiment. 

Consider  the  case  where  for  each  given  required  weapon,  the  correct 
assignment  and  each  of  the  two  incorrect  assignments  are  made  equally 
oiten.  Thus,  p^  (L)  =  1/3,  p,  (H)  =1/3  and  p^  (N)  =  1/3,  and  similar  re¬ 
sults  occur  for  p^  (L),  pj_j  (H),  etc.  Assuming  that  unequivocal  information 
transmission  would  have  yielded  the  correct  decision,  then  the  information 
transmitted  by  this  display  can  be  shown  to  be: 
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log*  3  -  1/3  log  3  -  1/3  log  3  -  1/3  log  3  »  0 

Here,  the  operator  Is  merely  guessing  and  the  display  conveys  no  informa¬ 
tion  to  him.  This  situation  is  diagrammed  below: 


p (L)  =  1/3  L 


p (H)  =  1/3  H 


p  (N)  =  1/3 


N 


i/3 


N 


Conditional  probabilities  are  shown  on  the  lines  joining  x,  the  wea¬ 
pon  which  should  be  assigned,  and  ^  the  weapon  which  is  assigned. 


L 


H 


The  results  for  a  second  display  wherenowthe  correct  assignment 
is  made  with  a  probability  of  1/2  and  each  of  the  incorrect  assignments  is 
made  with  probability  of  1/4  follows: 


p (L)  =  1/3 


p (H)  =  1/3 


p  (N)  =  1/3 


L  .  »/*  .  L 


The  information  conveyed  by  this  display  is 

log  3  -  1/2  log  2  -  1/4  log  4  -  1/4  log  4 
=  1.  58  -  1/2  -  1/4  •  2  -  1/4  .  2  =  1.  58  -  1.50  =  .08 

* 

All  logarithms  are  to  the  base  2. 
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This  means  that  out  of  each  158  bits  of  input  information  150  bits  are  lost 
because  of  the  "equivocation"  of  the  display.  Eight  bits  are  actually  con¬ 
veyed  to  the  operator. 

This  same  type  of  analysis  can  be  applied  to  battery  assignments. 
Consider,  for  example,  that  there  are  six  batteries  and  that  each  is  assign¬ 
ed  with  equal  probability.  The  input  information  is  then  log  6  =  2.58.  Sup¬ 
pose  that,  in  the  experiment,  it  is  found  that  the  correct  assignment  is 
made  3/4  of  the  time,  but  that  the  two  adjacent  batteries  are  assigned  1/8 
of  the  time.  The  information  conveyed  by  this  display  is: 

log  (6)  -  3/4  log  4/2  -  1/8  log  8  -  1/8  log  8 

=  2.58  -  3/4  (log  4  -  log  3)  -  1/4  .  3  =  1.52 

Hence,  out  of  each  258  bits,  152  are  conveyed  to  the  operator  and  106  are 
lost  because  of  the  ineffectiveness  of  the  display.  This  situation  is  shown 
below,  when  the  conditional  probabilities  are  sketched  in  for  batteries  D 
and  F. 


A 

B 

C 

D 

E 

F 


Thus,  when  F  should  have  been  assigned,  we  assume  that  it  was  found  that 
F  was  assigned  3/4  of  the  time  but  either  E  or  A  (the  adjacent  batteries) 
were  each  assigned  1/8  of  the  time. 

The  more  general  case  brings  Bayes  theorem  more  in  evidence,  i.  e.  , 
the  case  where  the  assignments  are  not  made  symetrically.  For  example, 
in  the  case  of  pictorial  displays,  certain  biases  can  be  expected  in  certain 
directions.  This  will  result  in  unsymmetric  conditional  probabilities. 

Let  the  correct  answers  have  probabilities  p(X)  in  a  given  series  of 
experiments, while  the  answers  given  by  the  subjects  have  probabilities  p(Y). 
This  is  shown  in  a  diagram  as  follows: 
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1. 

2. 

3. 

4. 


p  (Y) 


This  diagram  shows  that  when  3  should  have  been  assigned  2  was  actually 
assigned.  The  diagram  also  shows  the  conditional  probability  pX(Y)  that 
when  X  was  correct  Y  was  assigned. 

The  following  shows  an  actual  experimental  result  for  a  weapon  as¬ 
signment  problem  with  a  given  display. 


p(X) 

p  (Y) 

L 

1/3 

11/30 

H 

1/2 

13/30 

N 

1/6 

6/30 

Thirty  tests  were  run  with  this  display.  The  correct  assignment  in  10  of 
the  30  should  have  been  the  assignment  of  a  low  explosive  weapon,  but  this 
weapon  was  actually  assigned  11  times.  Hence,  there  was  one  incorrect 
assignment  of  a  low  explosive  weapon.  The  high  explosive  weapon  should 
have  been  assigned  15  out  of  the  30  times,  but  it  was  actually  only  assigned 
13  times.  The  nuclear  warhead  should  have  been  assigned  only  five  out  of 
30  times  but  it  was  actually  assigned  in  six  out  of  the  30  tests.  Hence,  it 
was  also  used  incorrectly  once  cut  of  the  30  tests.  The  following  shows 
the  situation  more  completely  because  in  this  diagram  it  is  actually  seen 
that  the  incorrect  assignments  should  have  been  made  to  the  high  explosive 
warhead: 


p(X) 

L  10/30  =  1/3 
H  15/30  =  1/2 
N  5/30  =  1/6 


p  ( Y) 
11/30 

13/30 

6/30 
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The  diagram  shows  the  conditional  probabilities.  The  (L)  =  1  and 
Pj^(N)  =  1  since  in  the  tests  with  this  display,  these  warheads  were  always 
assigned  correctly.  However,  when  H  was  the  input  in  15  tests,  it  was  as¬ 
signed  correctly  13  times  (p^(H)  =  13/  15),  L  was  assigned  once  in  the  15 
tests,  and  N  was  assigned  once  in  15  tests.  Hence  pjj(L)  =  1/15.  It  is  thus 
seen  how  the  diagram  is  derived  from  the  test  results. 

Bayes  formula;  p(X,  Y)  =  p(X)  p^  (Y),  is  now  used  to  derive  the  fol¬ 
lowing  table: 


p  (X,  Y) 

X,  Y  p(X,  Y)  p(X)  p ( Y) 


L, 

L 

1/3 

30/11 

H, 

L 

1.  30 

'  2/11 

H, 

H 

13/30 

2 

H, 

N 

1/30 

1/3 

N. 

N 

1/6 

5 

In  general,  this  table  for  this  problem  would  have  nine  entries,  but  four  of 
them  did  not  occur  in  these  tests,  i.  e.  ,  (L,  H),  (L,  N),  etc. 

These  are  all  the  data  required  to  calculate  the  effectiveness  of  the 
display  as  determined  by  these  test  results. 

1.  The  input  information  to  the  display  is 

-  Z  p(X)  log  p(X) 

| 

=  1/3  log  1/3  -  1/2  log  1/2 

-  1  / 6  log  1/6  =  1. 46 

2.  The  Information  transferred  by  the  display 
to  the  operators  is 

Z  p(X,  Y)  log  p  (X,  Y) 

p  (X),  P  ( Y) 

=  1/3  log  30/11  +  1/3  log  2/11  +  13/30  log  2 
+  1/30  log  1/3  +  1/6  log  5  =  1.  17 
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3,  The  information  lost  because  of  the  ambiguity 
of  the  display  =  input  information  -  informa¬ 
tion  transferred  =  1. 46  -  1.17  =  .29 

Thus,  out  of  every  146  bits  of  information  using  this  display,  117 

are  transferred  and  29  are  lost.  This  display  _LL_  =  .  80,  or  80%  efficient. 

146 

The  above  logic  was  applied  to  the  display  decision  data  derived  in 
the  present  study.  The  information  transferred  for  decision  22,  a  weapon 
launch  decision,  and  for  decisions  23  and  24,  battery  assignments  is  pre¬ 
sented  in  Table  9. 


Table  9 

Information  Transferred  (in  Bits)  for  Various  Decisions 


Decision 


Display  Type 


Bits  Transferred 


22  Pictorial-plus-Color  1.46 

Pictorial  Black-and-White  1.23 

Tabular- plus-Color  1.46 

Tabular  Black-and-White  1.16 

23  Pictorial-plus-Color  .  43 

Pictorial  Black-and-White  .  99 

Tabular- plus-Color  1.25 

Tabular  Black-and-White  1.  10 

24  Pictorial-plus-Color  1.85 

Pictorial  Black-and-White  1.76 

Tabular-plus-Color  1.88 

Tabular  Black-and-White  2.  07 


These  data  suggest  from  one  point  of  view,  that, if  anything,  the 
tabular  displays  were  more  efficient  than  the  pictorial  displays.  It  seems 
that  the  pictorial  display  gains  its  advantage  from  its  r elationistic  presenta¬ 
tion,  not  from  its  economy  of  information  transfer.  It  will  be  noted  that 
both  types  of  display  were  relatively  efficient  for  decisions  22  and  24  The 
advantage  of  color  coding  also  becomes  apparent,  once  again,  in  the  current 
analysis. 
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CHAPTER  III 


CROSS  VALIDATION 


While  the  results  of  experiment  I  suggested  the  validity  of  the  DEI 
technique  for  predicting  decision  ability  with  various  types  of  displays,  it 
was  considered  important  to  check  the  findings  in  an  independent  experi¬ 
ment.  Moreover,  while  experiment  I  considered  two  decision  levels,  the 
higher  or  more  difficult  decision  level,  which  involved  such  mental  aspects 
as  reasoning,  memory,  comprehension,  and  computation,  was  of  greater 
interest  than  the  lower  decision  level,  which  emphasized  the  ability  to  ex¬ 
tract  specific  units  of  information  from  a  display. 

In  experiment  II,  the  stimuli  remained  the  same  as  in  experiment  I. 
The  same  four  display  types  were  again  employed:  pictorial,  pictorial-plus- 
color,  tabular,  and  tabular- plus-color.  Also,  the  same  six  command  and 
control  situations  were  again  employed.  However,  one  concern  of  experi¬ 
ment  II  was  to  emphasize  the  higher  order  decision  processes.  Accord¬ 
ingly,  the  questions  answered  by  the  subjects,  which  involved  information 
extraction, were  eliminated.  Additional  questions  of  the  more  complex  de¬ 
cision  making  type  were  written  into  the  experiment. 

The  number  of  decisions  contained  in  each  problem  varied  as  a  func¬ 
tion  of  the  number  of  threats  in  a  problem.  The  total  number  of  decision¬ 
making  questions  for  all  six  problems  was  102.  For  the  less  saturated 
(fewer  threats),as  few  as  12  decisions  were  involved,while  the  most  heavily 
saturated  problems  had  as  many  as  25  decisions.  The  decision-making 
questions  used  in  experiment  II  appear  in  Appendix  E. 


Subjects 

Again,  twenty  male  college  students  served  as  subjects.  All  were 
students  at  either  Eastern  Baptist  College  or  Villanova  University.  None 
of  the  subjects  from  experiment  I  participated  in  experiment  II. 


Procedure 


The  subjects  were  randomly  assigned  to  one  of  the  four  experi¬ 
ment;,!  groups--one  group  for  each  presentation  technique.  No  one  group 
possessed  more  than  three  subjects  frorr.  a  specific  college. 

The  pay  scale  was  the  same  as  in  the  first  experimen*,  $1.  50  per 
hour.  However,  in  the  second  experiment,  the  subjects  were  not  offered 
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a  bonus  for  reaching  a  criterion  level  of  performance.  Instead,  they  were 
allowed  to  place  bets  on  their  battery  assignment  and  weapon  launch  de¬ 
cisions.  ms  a  result  of  these  bets,  they  could  win  up  to  six  additional  dol¬ 
lars.  The  betting  scheme  and  procedure  are  explained  more  fully  below. 

Virtually,  the  same  procedure  was  used  in  both  experiments.  How¬ 
ever,  certain  changes  were  introduced  in  order  to  accommodate  the  more 
extensive  probability  analysis  that  was  performed  on  the  data  resulting  from 
experiment  II.  Consequently,  during  the  second  session,  and  after  the  com¬ 
plete  retraining  and  remedial  training,  the  subjects  were  instructed  on  the 
betting  procedure  they  were  to  use  after  reaching  each  weapon  launch  and 
battery  assignment  decision. 

The  following  instructions  were  read  to  the  subjects  while  they  read 
them  to  themselves. 

DIRECTIONS  FOR  USING  PAYOFF  TABLES 


In  answering  questions  pertaining  to  battery  assignments 
and  weapon  launch  you  will  be  required  to  do  two  things,  First, 
you  will  make  the  assignment;  second,  you  will  make  a  bet.  You 
have  sixteen  cents  to  bet  on  each  question.  If  you  are  very  con¬ 
fident  that  the  assignment  you  made  is  correct,  then  you  may  bet 
the  entire  sixteen  cents  on  that  assignment.  However,  you  don't 
have  to  bet  the  whole  sixteen  cents  on  any  one  battery.  For  ex¬ 
ample,  if  you  think  that,  it  is  equally  likely  that  the  correct  choice 
might  be  either  of  two  batteries,  then  you  would  bet  eight  cents 
on  each  of  two  batteries.  Or,  if  you  think  that  the  correct  choice 
might  be  either  one  of  two  batteries  and  yon  think  that  the  chances 
are  3:1  that  the  correct  choice  is  one  battery  over  the  other,  then 
you  would  bet  twelve  cents  on  one  battery  and  four  cents  on  the 
other.  You  can  distribute  your  sixteen  cents  in  any  way  you  want 
over  the  eight  batteries  in  the  complex.  However,  the  total  six¬ 
teen  cents  must  be  committed  on  each  assignment,  be  it  battery 
assignment  or  weapon  launch. 

The  answer  sheets  you  will  be  given  have  spaces  in  which 
you  are  to  mark  your  answers  and  your  bets.  If  you  divide  your 
bet  over  more  than  one  battery  or  weapon  type,  you  indicate  this 
by  entering  tne  amounts  of  your  bet  in  the  appropriate  spaces. 

The  payoff  tables  show  the  amount  you  can  win  if  you  bet 
the  entire  sixteen  cents  on  a  given  battery  or  weapon  type  and 
that  battery  or  weapon  is  the  correct  choice.  Since  the  payoff 
(the  amount  you  can  win)  is  a  direct  function  of  the  payoff 
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table,  you  can  determine  how  much  you  would  win  by  taking 
the  proportion  of  sixteen  cents  that  you  bet  on  the  correct 
choice,  and  then  multiply  the  figure  shown  in  the  table  by 
the  same  proportions.  For  example,  if  battery  A  was  as¬ 
signed  when  it  should  have  been  and  you  bet  the  entire  six¬ 
teen  cents  you  would  win  fifty  cents,  If  instead,  you  bet 
four  cents  on  battery  A  when  it  should  have  been  assigned, 
then  you  would  win  1/4  of  fifty  cents  or  thirteen  cents.  If 
you  assigned  battery  D  when  battery  A  was  the  correct 
choice  and  you  bet  eight  cents  on  battery  D,  then  you  would 
win  1/2  of  twenty  cents  or  ten  cents. 

Money  will  not  be  on  the  for  every  assignment 
you  have  to  make.  However,  since  you  will  not  know  for 
which  assignments  money  is  on  the  line,  you  should  treat 
each  assignment  as  though  money  were  on  the  line.  You 
can  win  up  to  one  dollar  for  each  command  and  control 
display  you  are  shown,  in  addition  to  your  base  pay. 

Since  you  will  be  questioned  on  six  different  command  and 
control  displays,  you  can  win  up  to  six  dollars.  You  will 
be  asked  questions  other  than  battery  assignment  and  wea¬ 
pon  launch.  As  far  as  the  other  questions  are  concerned, 
you  can  win  three  cents  for  each  correct  answer.  In  no 
case  will  you  lose  any  money. 


All  questions  were  answered  fully  after  the  instructions  were  read 
and  then  the  experiment  began. 

The  pa}  off  table  was  arranged  to  yield  a  decreasing  payoff  as  the 
assignment  made  got  further  from  the  assignment  that  should  have  been 
made. 


Money  never  changed  h.  nds  during  the  experiment.  The  sixteen 
cents  the  subjects  had  to  bet  on  each  weapon  launch  and  battery  assignment 
decision  was  a  bet  on  paper  only.  At  the  conclusion  of  the  experiment,  the 
winnings  were  calculated. 

Two  sessions  were  held  with  each  group.  The  first  session,  in 
which  the  subjects  were  introduced  to  the  problem  and  the  initial  training 
that  was  presented  lasted  two  hours.  The  second  session  which  involved 
retraining  and  data  collection  took  approximately  four  hours. 

The  presentation  sequence,  and  exposure  intervals,  for  the  stimuli 
remained  the  same  as  in  experiment  I. 
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Results  and  Discussion 


The  subjects'  decision  scores  were  determined  and  sorted  accord¬ 
ing  to  the  variables  of  interest,  display  type  and  threat  numerosity  within 
display  type.  Since  there  was  a  different  number  of  decisions  to  be  made 
in  the  various  problems,  the  decision  scores  were  converted  into  per¬ 
centages.  The  data  were  subsequently  subjected  to  several  analyses,  as 
discussed  below. 

An  analysis  of  variance  was  performed  on  the  decision  scores. 
However,  because  the  values  were  now  expressed  in  terms  of  proportions, 
the  proportions  were  transformed  into  angles  or  arcsins  (Walker  and  Lev, 
1953)  for  the  variance  analysis. 

Table  10,  the  summary  table  of  the  variance  analysis,  indicates  a 
statistically  significant  difference  among  display  types  (p  <.  01).  Applica¬ 
tion  of  Tukey's  test  for  significant  differences  among  treatment  means  re¬ 
sulted  in  significant  differences  among  the  pictorial-plus-colot  (with  the 
highest  mean  decision  score)  and  both  tabular  displays,  and  between  the 
pictorial  black-and-white  and  the  tabular  black-and-white.  However,  the 
two  tabular  displays  did  not  differ  significantly  from  each  other  and  the 
pictorial  black-and-white  did  not  differ  significantly  from  the  tabular-plus- 
color.  Figure  9  presents  the  mean  decision  scores  for  each  display  type. 
Here,  an  almost  perfect  linear  relationship  between  decision  score  and 


Table  10 

Summary  of  Analysis  of  Variance  for  Experiment  II 


Sum  of  Degrees 


Source  of  Variation 

Squares 

of  Freedom 

Mean  Square 

F 

Problems  (P) 

.  63 

5 

.13 

1.18 

Display  Types  (DT) 

1  .  2n 

3 

.  43 

4.  00* 

P  x  DT 

2  09 

1  5 

.  14 

1 .  27 

Error 

10.  24 

96 

.  1  1 

TOTAL. 

14.  25 

119 

display  type  is  evidenced.  With  each  of  the  two  major  display  types,  the 
effect  of  color  was  to  enhance  the  decision  score.  When  color  was  added 
to  ‘.he  pictorial  display  type  there  was  an  increase  of  six  percent  in  the 


* 

Significant  at  or  below  Ihe  .  01  level  of  confidence 
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ME%K  DECISION  SCORE  ( TRANSFORMED  INTO  ARCSINS) 


Pictorial  pictorial  tmuui  taiuua 

PLUS  COLOR  SLACK  A  WHITE  PIUS  COLOR  HACK  I  WHITE 
OISPLAY  TYPE 


FIGURE  9  DECISION  SCORE  AS  A  FUNCTION  OF  DISPLAY  TYPE 


decision  score.  Similarly,  when  color  was  added  to  the  tabular  display 
type  there  was  an  increase  of  sevt-n  percent.  The  plot  also  shows  the  re¬ 
lative  superiority  of  the  pictorial  displays  over  the  tabular  displays,  for 
the  decisions  involved.  The  effects  of  color  and  the  effects  of  the  two 
major  display  types  (disregarding  color),  pictorial  and  tabular,  upon  de¬ 
cision  scores  are  indicated  for  each  stimulus  situation  in  F'iguros  10  and 
11. 


With  the  exceptions  of  stimulus  1  (an  ’  easy"  problem)  and  stimulus 
5  (a  "difficult"  problem)  in  Figure  11,  color  was  always  a  facilitating  agent. 
The  mean  (arcsin)  decision  score  for  color  across  all  problems  was  2  04, 
whereas  the  mean  (arcsin)  decision  score  for  the  black-and-white  displays 
was  1.91. 

In  no  instance  did  the  tabular  displays,  including  both  color  and 
black-and-white,  yield  higher  decision  scores  than  the  pictorial  displays, 
including  both  color  and  black-and-wh’te  (Figure  11).  The  pictorial  displays 
had  an  overall  mean  decision  score  (arcsin)  of  2.  08,  while  the  tabular  dis¬ 
plays  had  an  overall  arcsin  mean  decision  score  of  1.  87. 

Generally,  the  stimulus  situations  possessing  the  greater  number  of 
threats  (higher  numbered  stimulus  situations)  yielded  lower  decision  scores, 
a  finding  in  conformity  with  the  results  of  experiment  I. 


Application  of  the  DEI  Technique 

The  Display  Evaluative  Index  technique  was  applied  to  the  six 
stimulus  situations  and  four  display  types,  resulting  in  24  DEI  values.  The 
resulting  DEIs  are  presented  in  Table  11.  The  values  are  ranked  hierar¬ 
chically  from  high  to  low  in  both  rows  and  columns.  From  Table  11  the  ef¬ 
fect  of  stimulus  situation  (problem)  and  display  type  can  be  seen  from  the 
point  of  view  of  the  DEI  technique.  The  "easy"  problems,  with  fewer 
threats,  yielded  higher  values  than  the  more  dense  displays  or  "difficult" 
problems.  This  finding  is  in  accordance  with  the  empirically  derived  de¬ 
cision  scores  described  above.  Similarly,  the  pictorial  displays  were 
ranked  above  the  tabular  displays,  and  within  each  display  type  the  color 
displays  were  evaluated  as  being  superior  to  their  black-and-white  counter¬ 
parts.  These  findings  are  also  in  accordance  with  the  objective  decision 
scores . 


Correlational  Analyses 


The  relationship  between  the  ca’culated  DEIs  and  the  empirical  re¬ 
sults  of  the  experiment  was  calculated.  Pearson  product  moment 
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MEAN  DECISION  SCORE  (TRANSFORMED  INTO  ARCSINS) 


correlation  coefficients  were  oMimif  'l  b*-tw»'»n  the  decision  scores  am  th« 
PEI  values  for  each  display  type  within  each  problem  and  across  alt  prob¬ 
lems.  The  results  of  the  correlational  analyses  are  presented  in  Table  12 


Table  1  1 


PEI  Rank  Order  of  Problems  and  Display  Types 


Problem 

Pictorial- 
plus-Col  or 

Pictorial 
Black-  and -White 

Tabular- 

plus-Color 

Tabular 

Black-and-Whit 

1 

.  285 

.  248 

.  236 

.  232 

2 

.  253 

.  233 

.  215 

.  205 

3 

.  179 

.  154 

.  139 

.  133 

4 

.  132 

.  123 

.  098 

.  097 

5 

.  123 

.111 

.  098 

.  097 

6 

.  088 

.  080 

.  068 

.  065 

Table  12 


Product 

Between 

Moment  Correlation  Coefficients 
DEI  Values  and  Decision  Scores 

Problem 

Correlation 

£ 

Overall 

.  54 

<.01 

1 

.  44 

<.05 

2 

.  06 

— 

3 

.  45 

<  .  05 

4 

.  40 

---- 

5 

-.  01 

— 

6 

.  31 

— 

From  Table  12  it  can  be  seen  that  for  all  display  types  and  across 
all  problems,  an  adequate  relationship  (r  =  .  54)  was  found  to  exist  between 
the  DEI  and  the  decision  scores.  The  significant  level  for  th.s  correlation 
coefficient  (p  <  .  01  )  allows  the  null  hypothesis  (r  =  0)  to  be  rejected  and, 
hence, the  acceptance  of  the  hypothesis  that  a  positive  correlation  did,  in 
fact,  exist  between  the  DEI  values  and  the  decision  scores. 
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Confidence  or  Reasonableness  of  Ratting  Patterns 


The  subjects  in  this  experiment  were  asked  to  decide  upon  the  wea¬ 
pon  and  battery  to  assign  to  each  of  a  certain  number  of  threats.  Further¬ 
more, they  were  to  distribute  a  total  of  16  cents  in  bets  over  the  various 
weapon  and  battery  assignments,  for  each  threat.  It  was  strongly  hinted 
that  the  subjects  would  do  besi  if  they  distributed,  their  bets  in  proportion 
to  the  probabilities  of,  or  their  confidence  in,  their  choice.  In  reality, 
while  this  may  not  have  been  the  optimal  strategy,  it  had  one  advantage, 
viz.  ,  it  might  minimize  the  variation  in  the  payoff  for  each  trial.  As  will 
be  seen  later,  the  maximum  expected  payoff  is  obtained  by  always  betting 
all  the  money  (16  cents)  on  the  most  likely  event  (that  with  highest  prob¬ 
ability),  if  that  is  known.  Then  on  any  particular  trial,  the  payoff  is  maxi¬ 
mum  (50  cents)  and  the  average  (expected  value)  is  highest.  If  a  wrong 
guess  resulted  in  zero  payoff,  then  to  minimize  the  range  of  values  it  would 
be  necessary  to  distribute  the  bet  over  all  the  possibilities.  By  spreading 
the  bet  over  several  reasonable  possibilities  a  lower  bound  can  be  set  on 
the  gain  at  each  trial.  Therefore,  the  total  gain  is  at  least  n  times  this 
lower  bound,  where  n  is  the  number  of  trials.  The  latter  strategy  is  a  con¬ 
servative  one. 

To  measure  deviations  from  this  strategy,  the  following  formula  was 
devised: 


R  =  1-1/2  Z|p.  -  b.| 

where  R  might  be  called  the  reasonableness  or  rationality  of  the  subjects 
in  following  a  conservative  strategy  pj  is  the  probability  or  relative  cer¬ 
tainty  in  the  mind  of  the  subject  that  weapon  or  battery  i  is  the  correct  one, 
and  b|  is  the  fractional  part  of  the  total  amount  (16  cents)  that  is  placed  on 
choice  i.  If  the  distribution  of  the  total  amount  of  money  is  in  proportion 
to  the  probability  of  the  events,  then  p.  =  b.  for  all  i. 

Then  Pj  —  bj  =  0  for  i  and  R  =  1.  A  few  illustrations  are  given  in 
the  example  below.  The  example  on  the  first  line  assumes  it  to  be  equi- 
probable  (p  =  .  5)  that  battery  D  or  E  should  be  chosen  (is  the  correct  one). 
The  money  is  equally  distributed  over  these  possibilities  and  R  =  1,  In  ex¬ 
amples  5  and  6,  no  money  is  placed  on  possible  choices  (p^  >0).  These 
would  represent  cases  of  entire  lack  of  reasonableness  or  lack  of  rationality 
so  that  R  should  oe  zero. 
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Ex¬ 

A 

R 

C 

D 

E 

F 

G 

H 

z  lpi  ■  bil 

ample 

ElJl 

£^b 

XL* 

jKb 

Li 

jLk 

R 

i 

0,0 

0,0 

0,0 

•  5,0 

.5, .5 

0,0 

0,0 

0,0 

0 

1 

2 

0,0 

0,0 

»,» 

0,0 

0,0 

0,0 

0,0 

0,0 

0 

» 

3 

0,0 

•2,  .2 

•3.-3 

•3, -3 

•2, -2 

0,0 

0,0 

0,0 

0 

1 

4 

.1,0 

•3. -2 

.4, .3 

.2,  .4 

0,.l 

0,0 

0,0 

0,0 

.< 

•  7 

5 

Q,.2 

•  3,0 

.2,0 

0,-3 

0,  .4 

.2,0 

0,.l 

•  3,0 

2 

0 

6 

0,0 

0,0 

1,0 

0,1 

0,0 

0,0 

0,0 

0,0 

2 

0 

7 

0,0 

.  1 ,  .2 

•  I ,  .5 

•  £  >  •  z 

0,0 

.2,0 

o,.t 

0,0 

1.2 

.4 

8 

0,0 

0,0 

0,0 

0,0 

0,0 

I..5 

0..4 

0,0 

.8 

.6 

Examples  2  and  6  illustrate  the  cases  most  often  encountered  in  the 
present  project.  In  example  2,  the  subject  placed  16  cents  on  C  which  was 
the  correct  battery  (pc  =  1),  R  =  1.  In  example  6,  the  subject  place  16  cents 
on  battery  D  which  was  wrong  (p^  =  0)  so  R  =  0,  If  subjects  never  split  their 
bets,  then  R  would  be  either  0  or  1. 

So  far,  the  payoff  function  has  not  entered  into  the  computation.  The 
implicit  assumption  was  that  it  is  reasonable,  i.  e.  ,  that  the  reward  for  the 
correct  answer  is  maximum  and  '.hat  for  incorrect  answers  the  reward  is 
less  than  maximum  or  ever,  zero.  The  latter  is  usually  the  case. 

In  the  present  project  the  subjects  were  shown  the  payoff  table  be¬ 
fore  the  trials  began.  The  payoff  was  50  cents  for  a  correct  answer,  if  all 
16  cents  was  bet  on  that  choice,  or  an  amount  proportional  to  the  actual  bet. 
Thus,  if  12  cents  was  bet  on  the  correct  answer,  the  payoff  was  12/16  x  .50 
=  38  cents.  If  the  total  16  cents  was  bet  on  a  wrong  battery,  adjacent  to  the 
correct  one,  the  payoff  was  40  cents;  if  the  total  was  placed  on  a  wrong  bat¬ 
tery  two  positions  from  the  correct  battery,  the  payoff  was  30  cents,  and  so 
on.  This  payoff  distribution  might  tend  to  encourage  a  subject  to  bet  all  on 
the  most  probable  battery,  since  he  would  still  probably  obtain  a  reward  in 
a  particular  trial  in  which  this  battery  was  not  correct.  Suppose  the  payoff 
was  the  same  for  all  batteries,  correct  or  incorrect.  Then  any  answer 
would  yield  the  same  reward.  This  would  encourage  guessing  and  eliminate 
effort  on  the  part  of  the  subject.  The  present  triangular  distribution,  to  the 
extent  it  approaches  a  constant  payoff  function,  may  have  produced  a  similar 
effect.  This  may  be  justified  in  that  it  reduces  stress  in  the  subject  and  Still 
offers  some  incentive  to  do  well.  The  other  extreme  would  be  to  place  a 
penalty  (negative  payoff)  on  wrong  answers. 
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To  illustrate  the  effect  of  spreading  a  bet,  we  assume  the  following 
examples  with  the  batteries  under  consideration  limited  to  D,  E,  and  F: 


Example 

JL 

D 

b 

£_ 

E 

_ b 

E_ 

F 

_ b 

Expected 

Payoff 

Min. 

Payoff 

M  ax . 
Payoff 

R 

a 

.2 

.2 

.6 

.6 

.  2 

.  2 

43.  6 

40 

46 

1. 

b 

.  2 

0 

.  6 

1. 

.  2 

0 

46.  0 

40 

50 

.  5 

c 

.  3 

.  3 

.4 

.4 

.3 

.3 

41.  6 

40 

44 

1. 

d 

.  3 

0 

.4 

1. 

.  3 

0 

44.  0 

40 

50 

.  4 

e 

.  3 

.  3 

.4 

.  3 

.  3 

.4 

41.  2 

39 

41 

.  9 

f 

.  3 

.  2 

.4 

.  3 

.  3 

.5 

38.  2 

27 

43 

.  8 

The  payoff  is  given  by: 


Assigned 


D 

E 

F 

Should 

D 

50 

40 

30 

have 

E 

40 

50 

40 

assigned 

F 

30 

40 

50 

In  example  f,  R  =  8.  If  D  is  correct,  then  the  <  ::pccted  payoff  is 
50  bp,  !  1C  bp  i  30  bp  10  +  12  +  15  *  37  cents.  If  E  is  c^rrec*,  then  the 
payoff  is  40  bD  +  50  b£  +  40  bp  =  8  +  15  +  20  -  43  cents.  If  F  is  correct, 
then  the  payoff  is  30  bp)  +  40  bp;  +  50  bp  =  6  +  12  +  25  =  43  cents.  D  is  cor¬ 
rect  .  3  of  the  time,  E  is  correct  .  4  of  the  time,  and  F  is  correct  .  3  of  the 

time.  The  expected  value  of  payoff  when  the  bets  are  always  distributed  ir 
the  same  ratio  (2:3:5)  is 

E  =  pD  (27)  +  pE  (43)  +  pF  (43)  =  .  3  (27)  +  .4  (43) 

+  ,  3  (43)  =  8.  1  +  17.  2  +  12.  9  =  38.  2 


In  general, 

E  *  (50  b„  +  40  b_  +  30  b„)  +  p_  (40  b^  *  5"  b_ 

rD  D  E  F  E  D  E 

+  40  b_)  +  p_  (30  b__  +  40  bP  +  50  bj 
F  F  D  E  F 

'  50,PDbD. 

*  PF  bD  *  PE  bF  *  PF  bE  *  30  'PD  bF  +  PF  bD’ 
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In  exampie  f.  the  maxim'"”  pavnff  is  4  .'I  r  <-nts  (when  <ither  F  or  F 
is  correct)  and  the  minimum  gain  is  27  rents  (if  D  is  corrc  t),  I  x amples 
c  to  f  have  the  same  probability  distributions  but  varying  bet  distributions. 

In  example  c,  the  distributions  are  the  same  (conservative).  The  payoffs 
are  either  40  cents  or  44  cents  each  time,  with  the  expected  value  41,6 
and  R  -  1.  In  example  e  there  is  a  slight  change  in  the  bet  distribution 
among  E  and  F.  Minimum  and  maximum  payoffs  are  39  cents  and  41  cents. 
However,  the  expected  value  has  dropped  to  39  cents  and  R  to. 9.  The  best 
bet  distribution  is  in  example  4  with  all  on  E,  the  most  orobable.  The  mini¬ 
mum  and  maximum  values  are  now  40  cents  anc’  50  cents  while  the  expected 
value  has  reached  44  cents  which  is  the  maximum  attainable  for  any  bet 
distribution.  The  R  value  has  decreased  to  .  4  indicating  there  is  consider¬ 
able  deviation  from  the  equality  of  the  distributions. 

The  R  value  for  each  answer  for  each  subject  v'as  computed  through 
a  comparison  of  the  subject's  bet  distribution  with  what  the  subject  reported 
separately  as  his  answers  (regardless  of  the  correctness  of  the  answer). 

The  resultant  R  value  is  considered  to  reflect  the  subject's  confidence  in  his 
answer  and/or  the  reasonableness  of  his  bet  pattern. 

The  average  R  values  for  battery  assignment  are  given  below: 
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The  average  R  values  for  weapon  assignment  are  given  below: 
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These  results  indicate  that  most  of  the  subjects  were  consistent,  i.  e.  , 
they  placed  all  16  cents  on  the  possibility  they  thought  was  the  correct  one. 

In  view  of  the  nature  of  the  payoff  table  this  turned  out  to  be  the  most 
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reasonable  strategy.  However,  it  appears  that  the  subjects  were  also 
somewhat  overconfident.  The  decision  scores,  reported  earlier,  were 
not  accurate  in  many  cases.  There  is  no  indication  in  the  data  that  the 
subjects  were  more  reasonable  with  one  type  of  display,  than  with  another. 


Probabilitv  or  Information  Transfer  Analysis 


The  probability, or  information  transfer  analytic  method,  described 
in  the  last  section  of  Chapter  II,  was  applied  to  the  findings  of  the  second 
experiment.  The  results,  tabulated  across  subjects  and  decisions,  arc 


presented  in  Table  13. 


Table  13 


Information  Transferred  (Bits)  for  Each  Display  Type 


Problem  Type 


Bits  Transferred 


Battery  Assignment 


Pictorial-plus-Color 
Pictorial  Black-and-White 
Tabular-plus-Color 
Tabular  Black-and-White 


Weapon  Launch 


Pictorial-plus-Color 
Pictorial  Black-and-White 
Tabular- pi  us -Col  or 
Tabular  Black-and-White 


The  findings,  indicated  in  Table  13,  are  consistent  with  the  results 
of  the  first  experiment,  as  presented  in  Table  9.  The  battery  assignment 
tabulation  of  Table  13  is  based  on  a  total  of  27  decisions  per  subject  or  135 
decisions  per  display  type.  For  the  weapon  launch  tabulation,  33  decisions 
per  subject  were  involved.  Thus,  a  considerably  greater  number  of  de¬ 
cisions  is  represented  by  Table  13  as  compared  with  Table  9,  in  which  three 
decisions  per  subject  are  involved.  Even  with  the  increase  in  the  number  of 
decisions  involv'd  and  with  the  addition  of  a  risk  function  in  experiment  II, 
from  the  point  of  view  of  the  information  transfer  analysis,  no  significant 
superiority  is  found  for  the  pictorial  presentation.  As  in  experiment  I.  some 
value  is  indicated  in  the  use  of  redundant  color  coding.  Thus,  again,  it  seems 
that  the  effectiveness  of  the  pictorial  displays  s  not  so  much  in  their  effi¬ 
ciency  of  information  nr esentation,but  in  their  ability  to  help  the  user  to  deal 
with  relationships  in  context. 
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CHAP  I  I  H  IV 


DISCUSSION,  SUMMAKA,  AND  CONC1  USIONS 


The  principle  purpose  of  the  current  s<  t  of  studies  was  to  investi¬ 
gate  the  effectiveness  of  the  DEI  technique  for  predicting  the  effectiveness 
of  large  scale  displays.  The  findings  suggest  that  DEI  vaiues  for  a  set  of 
displays  do,  in  fact,  correlate  with  objective  decision  quality  scores  for 
decisions  made  on  the  basis  of  information  displayed  in  various  w?ys.  It 
was  also  indicated  that  the  DEI  technique  may  be  considered  to  be  a  better 
predictor  for  decision-making  scores  than  for  information-extraction  scores. 
As  the  result  of  this  tendency,  experiment  II  seemed  to  yield  greater  cor¬ 
relations  between  the  DEI  vaiues  and  the  objective  scores,  than  did  experi¬ 
ment  I,  The  principle  difference  in  experiment  II,  as  compared  with  experi¬ 
ment  I,  was  in  the  number  of  decision-making  questions  involved.  In  experi¬ 
ment  II,  no  information  extraction  decisions  were  involved, while  in  experi¬ 
ment  I,  each  subject's  total  decision  score  was  based  on  both  information 
extraction  and  decision-making  items.  A  superiority  in  the  ability  of  the 
DEI  technique  to  predict  the  decision-m aking,  as  opposed  to  the  information 
extraction,  situation  was  predictable.  The  DEI  technique  is  purported  to 
yield  an  evaluative  measure  of  the  display  reading  — >  decision  making  — * 
control  activation  links  in  a  man-equipment  system.  Thus,  the  DEI  tech¬ 
nique  was  designed  to  measure  certain  elements  of  the  decision-making  as¬ 
pects  of  man-equipment  operations. 

The  factors  in  the  pictorial  display  type  which  were  influential  in 
yielding  higher  DEI  values  were  the  lower  mismatch  factor,  the  lower  number 
of  elements,  the  fewer  links  involved.  The  tabular  displays  presented  infor¬ 
mation  to  three  significant  figures.  This  was  true  for  information  items  such 
as  range,  altitude,  and  range  rate.  However,  application  of  the  decision  rules 
to  make  a  battery  assignment  only  reouired  relative  judgment  of  distance  and 
general  knowledge  of  altitude  (greater  or  less  than  10,000  feet)  and  range  rate 
(inside  missile  range  at  present  or  within  15  seconds). 

Due  to  the  increased  simplicity  of  the  pictorial  type  display,  it  pos¬ 
sessed  fewer  links  and  it  required  fewer  information  transforms  and  con¬ 
versions.  Thus,  the  pictorial  displays  received  a  better  directness  score, 
a  lower  complexity  value,  and  a  higher  data  transfer  factor  score. 

The  finding  which  indicated  superiority  for  the  displays  involving  re¬ 
dundant  color  coding,  at  least  at  the  higher  decision  making  level,  was  also 
to  be  anticipated  and  supports  and  extends  the  findings  nf  others  (Hitt,  1961; 
Smith,  Farquhar,  and  Thomas,  1965).  Smith,  Farquhar,  and  Thomas 
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found  that  color  added  to  the  ability  to  perform  counting  and  comparison 
operations  in  formated  (tabular)  displays.  Hitt's  results  indicated  color- 
coding  to  be  superior  to  other  types  of  coding  (alpha-numerical,  shape, 
etc,  )  for  several  types  of  tasks.  On  the  other  hand,  the  results  of  Silver, 
Land's,  and  Jones  (1965)  yielded  an  indication  that  for  the  redundant  color- 
situation,  a  color  redundant  by  fact  interaction  existed.  The  authors  fail 
to  explain  this  finding.  We  prefer  to  interpret  our  findings  as  supporting 
the  principle  within  the  DEI  which  stated  that  information  transfer  effective¬ 
ness  will  increase  if  provision  is  made  for  redundancy  of  information. 

In  experiment  I,  statistically  significant  differences  were  found 
between  the  decision  scores  resulting  from  high  and  low  density  (target 
numerosity)  situations.  These  differences  were  also  reflected  by  the  DEI 
values.  This  indicates  that  the  DEI  is  sensitive  to  difficulty  level.  This 
finding  further  supports  arguments  favoring  the  sensitivity  of  the  DEI.  It 
is  pointed  out  that  this  type  of  employment  of  the  DEI  technique  represents 
something  of  an  extension  of  the  technique,  since,  at  the  outset,  the  techni¬ 
que  was  not  purported  to  differentiate  between  difficulty  levels  within  a  dis¬ 
play  but  only  to  discriminate  between  different  display  designs.  The  finding 
that  as  fact  density  (threat  numerosity)  increased,  decision  scores  decreased, 
is  in  accordance  with  the  findings  of  previous  investigators  (Silver,  Landis, 
and  Jones,  1965).  However,  the  present  study  extends  this  previous  work  by 
pointing  out  that  increasing  fact  density  debi’itates  complex  decision-making 
to  a  greater  extent  than  it  debilitates  information  extraction. 

V/e  hypothesized  that  the  pictorial  displays  would  yield  better  scores 
for  decision  level  II,  involving  complex  dec  lion-making,  but  that  the  tabular 
displays  would  be  found  more  facilitative  for  the  extraction  of  simple  facts 
such  as  system  status,  availability  of  units,  absolute  quantitative  level,  and 
the  like.  The  results  of  experiment  I  tended  to  support  this  hypothesis.  The 
tabular  displays  were  superior  to  the  pictorial  displays  for  decision  level  I 
and  the  reverse  held  for  decision  level  II.  When  the  results  of  both  levels 
were  grouped,  the  pictorial  displays  seemed  superior.  In  the  second  experi¬ 
ment,  which  emphasized  decision  level  II,  the  pictorial  type  of  display 
yielded  higher  decision  scores.  This  finding  held  across  problem  difficulty 
levels  and  whether  redundant  color  was  c  -  was  not  involved. 

The  information  transfer  analyses  tended  to  point  out  that  different 
methods  will  tend  to  produce  different  (and  possibly  antithetical)  indications, 
regarding  the  utility  of  a  given  display  or  set  of  displays.  While  the  pictorial 
displays  seemed  to  produce  higher  decision  scores,  the  tabular  displays  ap¬ 
peared  to  be  more  efficient.  This  finding  seemed  to  hold  for  both  experiment 
I  and  experiment  II.  Thus,  it  seems  that  different  evaluative  methods  must 
be  used  in  any  display  evaluation  that  purports  to  be  complete.  If  decision¬ 
making  optimization  is  desired,  then  the  DEI  technique  results  will  probably 
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yield  the  required  information.  On  the  other  hand,  if  information  transfer 
effieiency  best  represents  the  required  result,  then  alternative  evaluational 
techniques  are  probably  needed. 

Our  analysis  of  the  reasonableness  of  the  betting  patterns  again 
points  up  the  importance  of  a  number  of  measurement  techniques  for  evalu¬ 
ating  displays.  The  betting  patterns  of  the  subjects  did  not  appear  to  differ 
across  the  display  types.  The  subjects  tended  to  place  their  total  bet  al¬ 
location,  for  any  single  trial,  on  tne  possibility  which  they  believed  to  be 
most  correct.  Hence,  the  rationality  of  the  betting  patterns  failed  to  dis¬ 
criminate  among  display  types.  However,  it  is  probably  a  truism  that  a 
display  which  yields  a  low  R  value  (which  we  interpret  as  an  index  of  sub¬ 
ject  reasonableness  in  employing  the  display  or  confidence  in  his  decision 
based  on  the  information  transferred  by  the  display),  will  be  les3  desirable 
than  a  display  which  yields  a  high  R  value. 

The  impetus  for  the  current  study  was  the  need  for  a  technique  for 
evaluating  large  scale  displays  while  the  displays  are  in  the  conceptual 
phase  of  development.  Previous  studies  established  the  validity  of  the 
technique  for  predicting  display  reading  information  processing  con¬ 
trol  action  situations  with  smaller  displays  of  other  natures.  The  results 
of  the  present  work  support  the  contention  that  the  techrique  possesses  ade¬ 
quate  power  to  discriminate  between  displays  of  different  designs  and  to 
predict  the  ability  of  persons  to  make  decisions  on  the  basis  of  the  displays 
when  large  scale  screen  displays  are  involved.  The  technique  is  objective, 
quantitative,  and  possesses  acceptable  reliability.  As  such,  it  seems  that 
the  DEI  technique  will,  at  least  partially,  fill  the  gap  which  motivated  the 
prerent  work.  It  is  not  our  contention  that  the  technique  is  a  substitute  for 
static  or  dynamic  mock-up  evaluations  or  other  human  factors  evaluative 
techniques.  We  do  maintain,  however,  that  the  technique  will  be  useful  for 
hierarchically  ordering  alternate  techniques  for  displaying  information  when 
the  various  techniques  are  in  the  conceptual  phase.  Quite  obviously,  the 
technique  does  not  consider  engineering  feasibility,  cost,  and  human  en¬ 
gineering  principles  of  the  check-list  variety.  Improvements  to  the  techni¬ 
que  remain  possible,  especially  from  the  information  exti  action  point  of 
view. 

Summary  and  Conclusions 

Two  experiments  were  performed  in  order  to  investigate  the  re¬ 
lationship  between  the  values  yielded  by  the  Display  Evaluative  Index  techni¬ 
que,  a  technique  for  evaluating  the  ability  of  displays  to  transfer  information 
to  the  operator  and  for  the  operator  to  act  on  this  information,  and  objective 
scoring  of  decisions  based  on  these  displays.  Other  purposes  of  the  work 


included:  (1)  comparison  of  the  effectiveness  of  pictorial  and  tabular  dis¬ 
plays  from  the  decision  correctness  and  the  information  extraction  points 
of  view,  (2)  investigation  of  the  advantages,  if  any,  of  redundant  color  cod¬ 
ing  in  a  display,  (3)  investigation  of  the  information  transfer  efficiency  of 
pictorial  and  tabular  displays,  (4)  investigation  of  the  effect  of  fact  density 
on  decision  scores,  and  (5)  investigation  of  the  reasonableness  of  the  betting 
pattern  or  the  confidence  produced  by  different  display  types. 

Four  display  types  were  prepared  and  the  same  six  command-control 
stimulus  situations  were  represented  on  each  display  type.  The  display  types 
were  pictorial,  tabular,  pictorial-plus-color,  and  tabular-plus-eolor.  The 
six  command-control  situations  represented  an  increasing  order  of  .act  den¬ 
sity  or  difficulty.  In  each  experiment,  a  separate  group  was  trained  in  read¬ 
ing,  interpreting,  and  applying  the  information  of  each  display  type.  Each 
group  v/as  then  exposed  to  the  display  type  on  which  it  had  been  trained  and 
its  ability  to  use  the  display  determined.  Experiment  I  differed  from  experi¬ 
ment  II  mainly  in  its  emphasis  on  obtaining  both  an  information  extraction 
and  a  decision-making  score.  In  experiment  II,  only  decision-making  scores 
were  involved.  Display  Evaluative  Index  technique  values  were  calculated 
separately  for  each  display. 

The  following  conclusions  seem  warranted  by  the  data: 

1.  The  Display  Evaluative  Index  (DEI)  technique 
values  were  adequately  predictive  of  the  ob¬ 
jective  decision-making  scores. 

2.  The  Display  Evaluative  Index  technique  values 
were  less  adequate  for  predicting  information 
extraction  scores. 

3.  Pictorial  displays  were  generally  superior  to 
tabular  displays  for  decision-making;  for  in¬ 
formation  extraction  this  superiority  did  not 
seem  to  hold. 

4.  Color  redundancy  seemed  to  increase  the  ability 
of  both  tabular  and  pictorial  displays  to  transmit 
information,  at  least  at  the  higher  decision  level. 

Adding  color  to  the  pictorial  displays  seemed  to 
exert  a  greater  positive  effect  than  did  the  addi¬ 
tion  of  color  to  the  tabular  displays. 
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5.  Although  the  pictorial  display  type  seemed  to 
produce  higher  decision  scores  than  the  tabular, 
the  efficiency  of  the  pictorial  display,  in  terms 
of  minimizing  information  equivocation,  seemed 
to  be  less  than  that  of  the  tabular  display. 

6.  Increasing  information  redundancy  increased  in¬ 
formation  extraction  scores. 

7.  Increasing  fact  density  yielded  lower  decision¬ 
making,  and  information  extraction  scores.  How¬ 
ever  the  effect  on  decision  making  scores  seemed 
to  be  greater  than  the  effect  on  information  extrac¬ 
tion. 

8.  In  terms  of  the  index  employed,  no  difference  was 
noted  between  the  betting  patterns  and  the  operator 
confidence  emerging  from  employment  of  the  four 
display  types  involved. 
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APPENDIX  A 

QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATIONS 

Experiment  I 
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QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATIONS 


1.  How  many  batteries  are  unassigned? 

2.  How  many  batteries  are  ready  for  assignment  but  not  yet  assigned? 

3.  How  many  batteries  in  the  Washington-Baltimore  complex  are  assigned? 

4.  How  many  batteries  in  the  Philadelphia  complex  have  launched  missiles? 

5.  Wh^!  i.e  the  total  numb**:  of  nuclear  weapons  that  have  been  !aur."h''d 
prior  to  the  present  situation? 

6.  What  is  the  total  number  of  high  explosive  weapons  available  for  the 
Washington-Baltimore  complex? 

7.  How  many  batteries  in  the  Washington-Baltimore  complex  have  more 
than  five  low  explosive  weapons? 

8.  How  many  batteries  in  the  Philadelphia  complex  hive  ten  or  more  hits? 

9.  How  many  threats  are  above  29,  999  feet? 

10.  How  many  kill  probabilities  have  been  computed  at  .  50  or  more? 

1  1.  How  many  threats  contain  between  2  and  9  targets? 

12.  How  many  unassigned  threats  are  inside  the  missile  ranges,  or  will  be 
inside  the  missile  ranges  within  the  next  fifteen  seconds? 

13.  What  is  the  total  number  of  unassigned  threats  in  the  northeast,  as  de¬ 
termined  from  each  complex? 

14.  How  many  threats  contain  nuclear  warheads? 

15.  How  many  threats  are  on  headings  of  between  181°  and  270°? 

16.  Which  type  of  weapon  has  been  launched  most  frequently  in  the  Phila¬ 
delphia  complex  prior  to  the  present  situation? 

17.  How  many  threats  are  at  actual  distances  of  175  miles  or  more  from  the 
Philadelphia  complex?  (Exclude  threats  assigned  to  the  Washington-Bal¬ 
timore  complex). 
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18.  How  many  threats  are  in  regions  of  rain? 

19.  How  many  threats  are  in  regions  where  the  winds  exceed  20  mph? 

20.  How  many  computers  are  ok? 

21.  Which  threat  should  be  assigned  next? 

22.  Which  weapon  should  battery _ launch  against  threat _ ? 

23.  To  whi  h  battery  should  threat _ most  probably  be  assigned? 

24.  To  which  battery  should  threat  _ most  probably  be  assigned? 

25.  What  is  the  overall  probability  of  kill  across  all  threats? 

26.  If  both  Philadelphia  computers  were  to  fail  now,  which  threat  would  you 
assign  first  to  the  Washington- Baltimore  complex? 

27.  If  both  Washington- Baltimore  computers  were  to  fail  now,  which  threat 
would  you  assign  first  to  the  Philadelphia  complex? 

28.  Which  battery(ies),  if  any,  selected  an  improper  missile  to  launch? 
(Answer  "None,  "  if  appropriate). 

29.  Assuming  a  wind  shift  of  180°,  for  which  threats  would  you  select  a  dif¬ 
ferent  missile  to  launch?  (Answer  "None,  "  if  appropriate), 

30.  Which  threat(s),  if  any,  should  be  assigned  to  more  than  one  battery? 
(Answer  "None,  "  if  appropriate). 
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APPENDIX  B 


CODES  FOR  PICTORIAL  AND  TABULAR  DISPLAYS 
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CODING  FOR  PICTORIAL-- BLACK  AND  WHITE  DISPLAYS 


Applied  Psychological  Services 
Science  Center 
Wayne,  Pennsylvania 


I.  Threat  Status 

A.  Threat  number: 

1.  Threat  numbers  range  from  01  to  15 

2,  If  threat  number  begins  with  the  numeral  5, 
the  threat  contains  a  nuclear  warhead,  e,  g.  , 
58 

B.  Battery  assigned: 

If  an  arc  (/  ^  )  appears  above  the  threat  number 
the  threat  has  been  assigned  to  a  battery 

C.  AltituJj. 

1.  *  =  0  -  14,  999  feet 

2.  I  =  15,  000  -  29.  999  feet 

3.  \  =  30,  000  -  49,  999  feet 

4.  •  =  50,  000  feet  and  over 

D.  Missiles  launched: 

1.  v  /  =  low  explosive  weapon 

2.  =  high  explosive  weapon 

3-  =  nuclear  weapon 
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E.  Actual  position  and  threat  size: 

1.  The  position  of  the  circle  indicates  the  actual 
or  present  position  of  the  threat,  as  measured 
in  miles  from  the  center  of  the  complex 

=  single  target  { 1 ) 

=  few  targets  (2  -  9) 

=  many  targets  (10  or  more) 

F.  Predicted  rang<_. 

Indicates  predicted  position  of  threat  after  15  seconds, 
as  measured  in  miles  from  the  center  of  the  complex 

G.  Directional  line: 

Indicates  relative  direction  in  which  threat  is  heading 

H.  Predicted  intercept  point: 

Indicates  the  point  at  which  the  threat  will  be  inter¬ 
cepted  by  the  battery  launched  weapon 

I.  Battery  assigned: 

Indicates  the  battery  assigned  to  the  threat 

J.  Missile  ranges: 

Indicates  maximum  range  of  the  missiles  used  by 
batteries-- radius  of  missile  range  equals  150  miles 

K.  Battery  designators: 

1.  Batteries  A  through  H  represent  the  eight  bat¬ 
teries  surrounding  t  le  Philadelphia  complex 

2.  Batteries  I  through  P  represent  the  eight  bat¬ 
teries  surrounding  the  Washington-Baltimore 
complex 


3.  O 

4, 
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L.  Weather  indicators: 


1. 

Q  =  clear 

2. 

®  -  rain 

3. 

(?)  *  snow 

4. 

-  wind  direction 

5. 

=  full  barb  equals  10  mpt 

,/t 

6. 

( J  =  1/2  barb  equals  5  mph 

M.  Comptex  designators: 

i  * 

1. 

=  Philadelphia 

2. 

=  Washington- Baltimore 

Battery  Status 

A.  ' 

=  ready  for  assignment 

B. 

=  assigned 

C>-}  M 

=  missile  type  launched 

D.  Kill  probability: 

1. 

o 

o 

li 

O 

2. 

O'  =  .25 

3. 

Q  =  .50 

4. 

-O  =  -75 

III.  Weapons  Available  and  Miaslle  Types  Launched 

A.  -  nuclear 

B.  =  high  explosive 

C.  y/^  =  low  explosive 
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CODING  FOR  TABULAR-- BLACK-AND-WHITE  DISPLAYS 


Applied  Psychological  Services 
Science  Center 
Wayne,  Pennsylvania 


I.  Threat  Status 


A.  Threat  number: 

1.  Threat  numbers  range  from  01  to  15 

2.  If  threat  number  begins  with  the  numeral  5, 
the  threat  contains  a  nuclear  warhead,  e.  g,  , 
58 

B.  Threat  size 

1.  $ 

2.  Rev' 

3. 

C.  Altitude: 

Indicates  altitude  of  the  threat,  in  feet 

D.  Actual  range: 

Indicates  the  actual  or  present  rang'  of  the 
threat,  as  measured  in  miles  from  the  center 
of  the  complex 

E.  Predicted  range: 

1.  Indicates  predicted  range  of  threat  after 

15  seconds,  as  measured  in  miles  from  the 
center  of  the  complex 

2.  The  missile  ranges  indicate  the  maximum 
range  of  the  missiles  used  by  the  batteries-- 
radius  of  missile  range  equals  150  miles 


=  1  target 

=2-9  targets 

=  10  or  more  targets 
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F.  Complex: 


Two  complexes  are  involved-- Phita  and  Wash; 
each  complex  consists  of  eight  batteries  and 
all  distance  measurements  arc  made  from  the 
center  of  the  complex 

G.  Battery  assigned: 

Indicates  the  battery  to  which  the  threat  is  as¬ 
signed;  batteries  A  through  H  are  the  eight 
batteries  surrounding  the  Philadelphia  complex, 
and  batteries  I  through  P  are  the  eight  batteries 
surrounding  the  Washington- Baltimore  complex 

H.  Missiles  launched: 

1.  L  =  low  explosive 

2.  =  high  explosive 

3.  N  -  nuclear 

I.  Actual  heading: 

Indicates  the  polar  heading  of  the  threat  in  de¬ 
grees,  as  measured  from  the  actual  or  present 
position  of  the  threat 

J.  Predicted  heading: 

Indicates  the  polar  heading  of  the  threat  in  de¬ 
grees,  as  measured  from  the  predicted  position 
of  the  threat  after  15  seconds 

K.  Actual  coor’d: 

Indicates  the  present  position  of  the  threat  in 
terms  of  compass  bearings,  e.  g.  ,  N,  NE,  E, 

SE,  S,  SW,  W,  and  NW 

L.  Predicted  coor'd: 

Indicates  the  predicted  position  of  the  threat  in 
terms  of  compass  bearings,  e.  g.  ,  N,  NE,  E, 

SE,  S,  SW,  W,  and  NW 
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M.  Intercept  point  range: 

Indicates  the  distance,  in  miles  from  the  center 
of  the  complex,  at  which  the  threat  will  be  inter¬ 
cepted  by  the  battery  launched  weapon 

N,  Intercept  point  coor'd: 

Indicates  the  position  of  the  intercept  point  in 
terms  of  compass  bearings,  e.  g.  ,  N,  NE,  E, 

SE,  S,  SW,  W  and  NW 


II.  Battery  Status 


A.  Coor'd  =  compass  bearings  of  the  batteries,  e.  g.  ,  N, 
NE,  E,  SE,  S,  SW,  W,  and  NW 


B.  Des  =  battery  designators-- A  through  H  and  I  through  P 

C.  ■  =  ready  for  assignment 

D.  Assigned  =  Y  =  yes,  V  =  no 

E. L,/y,/tf  r  missile  type  launched 
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APPENDIX  C 


GROUND  RULES  FOR  DECISION-MAKING 
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GROUND  RULES 


Applied  Psychological  Services 
Science  Center 
Wayne,  Pennsylvania 


I.  For  Selecting  the  Weapon  to  Launch 


A.  Nuclear  weapons  are  used  when  all  of  the  following  conditions 
exist: 

1.  The  wind  is  not  in  the  direction  of  the 
complex 

2.  The  threat  has  a  few  targets  (2  to  9)  or 
many  targets  (10  or  more) 

3.  The  threat  contains  a  nuclear  warhead 

4.  The  threat  is  above  14,  999  feet 

5.  The  intercept  point  is,  or  will  probably 
be,  more  than  50  miles  from  the  center 
of  a  complex 

B.  High  explosive  weapons  are  used  when  the  following  conditions 
exist: 

1.  The  threat  contains  a  few  targets  (2  to  9) 
or  many  targets  (10  or  more) 

2.  The  intercept  point  is,  or  probably  will  be, 
more  than  50  miles  from  the  center  of  a 
complex 

C.  Low  explosive  weapons  are  used  when  the  following  conditions 
exist: 

1.  The  threat  contains  a  single  target  and/o.' 

2.  The  intercept  point  is,  or  probably  will  be, 
within  50  miles  of  the  center  of  a  complex 
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II.  For  Assigning  Batteries  to  Threats 

A.  The  computer  status  must  be  o.  k.  for  at  least  one  computer  in 
a  complex  before  assigning  a  battery  in  that  complex  to  a 
threat. 

B.  Threats  are  assigned  to  the  nearest  unassigned  battery  with 
at  least  five  of  the  appropriate  weapon  type  available.  If 
there  are  :wo  batteries  equidistant  from  the  threat  and  each 
has  at  least  five  of  the  appropriate  weapon  available,  that 
battery  is  selected  that  has  the  highest  batting  average. 

Batting  average  is  determined  by  using  the  following  formula. 

_  ,  Hits 

Batting  Average  =  rr - rr - 

b  Misses  +  Aborts 

C.  Two  batteries  are  always  assigned  to  a  single  threat  when 
the  threat  is  within  30  miles  of  the  center  of  the  complex 
and  the  survival  probability  of  the  threat  is  .  75  or  greater 
(survival  probability  =1  —  kill  probability).  This  assign¬ 
ment  has  the  highest  priority--it  must  be  made  before  any 
other  assignment.  Moreover,  rather  than  the  nearest  unas¬ 
signed  battery,  the  choice  of  battery  to  receive  the  assign¬ 
ment  is  made  so  as  to  maximize  kill  probability.  Therefore, 
the  unassigned  battery  with  the  highest  batting  average  and 
with  at  least  five  of  the  appropriate  weapon  type  availabe  is 
selected. 

D.  When  the  kill  probability  is  not  given,  the  be3t  estimate  is 
.  50. 

III.  For  Assigning  One  Battery  to  More  than  One  Threat 

A.  A  battery  should  receive  an  assignment  of  more  than  one 
threat  if  all  the  other  batteries  are  either  assigned  or  not 
ready  to  receive  an  assignment. 

B.  The  batteries  to  receive  multiple  threat  assignments  are 
selected  on  the  basis  of 

1.  the  nearest  with  at  least  five  of  the  ap¬ 
propriate  weapon  type 

2.  the  highest  batting  average 


73 


C.  A  battery  that  has  launched  a  weapon  against  a  threat 
is  still  engaged  and  assigning  this  battery  to  another 
threat  is  considered  a  multiple  threat  assignment 

IV.  For  Determining  the  Weather  Conditions  Around  a  Threat 

Use  the  weather  forecast  closest  to  the  threat 

V.  For  Reassigning  Threats  when  Both  Computers  Fail 

If  both  computers  fail  for  either  complex,  only 
those  threats  that  are  within  the  missile  range 
of  the  other  complex  should  be  reassigned  to 
that  complex 


74 


APPENDIX  D 

SYMBOL  TEST  FOR  PICTORIAL  AND  TABULAR  DISPLAYS 
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SYMBOL  TEST  FOR  PICTORIAL— BLACK  AND  WHITE  DISPLAYS 


Applied  Psychological  Services 
Science  Center 
Wayne,  Pennsylvania 


Name 


Date 


1.  What  altitudes  are  indicated  in  the  following  codes? 
♦ 

A.  *.  Answer _ 

B.  ‘  Answer _ 

C.  •  Answer _ 

D.  J  Answer 


2. 


3. 


4. 


What  is  indicated  in  the  following  codes? 


A. 

' — \ 

Answer 

B. 

L/ 

Answer 

c. 

Answer 

D. 

Answer 

What  is  indicated  when  a  threat  number  begins  with  the  numeral  5? 
Answer _ 

What  is  represented  by  the  following  symbols? 


A. 

• 

Answer 

B. 

0 

Answer 

c. 

0 

Answer 
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5.  Which  of  the  following  symbols  indicates  (respond  with  the  appropriate 
letter) 


A.  an  assigned  threat?  Answer 

B.  actual  range?  Answer 

C.  predicted  range?  Answer 

D.  intercept  point?  Answer 


6.  Reproduce  the  appropriate  weather  symbols  in  the  space  provided. 

A.  Clear  Answer 

B.  Snow 

C.  Wind  speed  of  15  mph 


Answer 

Answer 


7.  Reproduce  the  appropriate  kill  probability  symbols  in  the  space  pro¬ 
vided. 


A. 

.25 

Answer 

B. 

.50 

Answer 

c. 

.75 

Answer 

D. 

1,00 

Answer 

8.  Which  one  of  the  following  symbols  represents  a  high  explosive  weapon? 


C. 


Answer 
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9.  What  is  represented  by  the  circles  around  the  Philadelphia  and  Wash¬ 
ington- Baltimore  complexes? 

Answer _ 

10.  According  to  the  Ground  Rules,  which  weapon  would  you  launch,  given 
the  following  situations? 


L 


Answer 
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SYMBOL  TEST  FOR  TABULAR- -BLACK  AND  WHITE  DISPLAYS 


Applied  Psychological  Services 
Science  Center 
Wayne,  Pennsylvania 


Name 


Date 


1.  What  is  indicated  when  a  threat  number  begins  with  the  numeral  5? 

Answer _ 

2.  What  do  the  following  symbols  indicate?  , 

A.  /V  Answer _ 

B.  H  Answer _ 

C.  L  Answer _ 

3.  What  is  indicated  by  the  "actual  range"  value? 

Answer _ 

4.  What  is  indicated  by  the  "predicted  range"  value? 

Answer _ 

5.  If  you  wanted  to  know  the  direction  in  which  a  threat  was  heading,  where 
would  you  refer? 

Answer _ 

6.  If  you  wanted  to  know  where  a  threat  was  located  in  reference  to  the  com¬ 
plex,  where  would  you  refer? 

Answer 


79 


7.  If  you  wanted  to  know  the  direction  in  which  a  threat  will  be  heading  in 
15  seconds,  where  would  you  refer? 


Answer 


8.  If  the  actual  coor'd  of  a  threat  is  given  as  NW  and  the  wind  direction  is 


given  as  follows 
(yes,  no) 


•  -0 


6  is  the  wind  in  the  direction  of  the  complex? 


Answer 

I  I 

9.  What  does  ^  indicate? 

Answer _ 

10.  Describe  the  state  of  the  battery  M  as  given  in  the  following  situation? 

Des.  Ready  Assigned 


/a  m  m 


Answer 


11.  According  to  the  Ground  Rules,  which  weapon  would  you  launch,  given 
the  following  situation? 


Threat  O  /  ,  few  ,  Range  Predicted 
Actual  Range  =  ,  Altitude  =  iS  O  t 

Heading  =  370  ,  Actual  Coor'd  =  £  , 

Heading  =  370  ,  Predicted  Coor'd  =  £ 
Point  =  /V  7  .  Intercept  Coor'd  =  £  , 


Answer 


=  ns 

i 

Actual 
Predicted 
,  Intercept 

Weather 
•J 

S 
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B. 


Threat  Si  ,  ,  Range  Predicted  =  I5S  , 

Actual  Range  =  0  .  Altitude  =  375  ,  Actual 


Heading  =  ,  Actual  Coor’d  =  t  ,  Predicted 

Heading  =  ,  Predicted  Coor’d  =  £  .  Intercept 

Point  =  US  ,  Intercept  Coor'd  =  t  ,  Weather  j £ 


£ 


l/vou/1 


Answer 


APPENDIX  E 

QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATIONS 

Experiment  II 
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QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATIONS  NO.  1 


1.  Which  weapon  should  be  launched  against  threat  11? 

2.  Which  weapon  should  be  launched  against  threat  09? 

3.  Which  weapon  should  be  launched  against  threat  53? 

4.  To  which  battery  should  threat  09  most  probably  be  assigned? 

5.  To  which  battery  should  threat  53  most  probably  be  assigned? 

6.  Which  threat  should  be  assigned  next? 

7.  What  is  the  overall  probability  of  kill  across  all  threats? 

8.  If  both  Philadelphia  computers  were  to  fail  now,  which  threat  would  you 
assign  first  to  the  Washington-Baltimore  complex? 

9.  If  both  Washington-Baltimore  computers  were  to  fail  now,  which  threat 
would  you  assign  first  to  the  Philadelphia  complex? 

10.  Which  battery(ies),  if  any,  selected  an  improper  missile  to  launch? 

11.  Assuming  a  wind  shift  of  180°,  for  which  threats  would  you  select  a  dif¬ 
ferent  missile  to  launch?  (Answer  "None,  "  if  appropriate). 

12.  Which  threat(s),  if  any,  should  be  assigned  to  more  than  one  battery? 
(Answer  "None,  "  if  appropriate). 
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QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATION  NO.  2 


1.  Which  weapon  should  be  launched  against  threat  12? 

2.  Which  weapon  should  be  launched  against  threat  53? 

3.  Which  weapon  should  be  launched  against  threat  06? 

4.  To  which  battery  should  threat  06  most  probably  be  assigned? 

5.  To  which  battery  should  threat  07  most  probably  be  assigned? 

6.  Which  threat  should  be  assigned  next? 

7.  What  is  the  overall  probability  of  kill  across  all  threats? 

8.  If  both  Philadelphia  computers  were  to  fail  now,  which  threat  would  you 
assign  first  to  the  Washington- Baltimore  complex? 

9.  If  both  Washington- Baltimore  computers  were  to  fail  now,  which  threat 
would  you  assign  first  to  the  Philadelphia  complex? 

10.  Which  battery(ies),  if  any,  selected  an  improper  missile  to  launch? 
(Answer  "None,  "  if  appropriate). 

1  1.  Assuming  a  wind  shift  of  180°,  for  which  threats  would  you  select  a  dif¬ 
ferent  missile  to  launch?  (Answer  "None,  "  if  appropriate). 

12.  Which  threat(s),  if  any,  should  be  assigned  to  more  than  one  battery? 
(Answer  "None,  "  if  appropriate). 
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QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATION  NO,  3 


1.  Which  weapon  should  be  launched  against  threat  57? 

2.  Which  weapon  should  be  launched  against  threat  51? 

3.  Which  weapon  should  be  launched  against  threat  54? 

4.  Which  weapon  should  be  launched  against  threat  09? 

5.  Which  weapon  should  be  launched  against  threat  12? 

6.  To  which  battery  should  threat  54  most  probably  be  assigned? 

7.  To  which  battery  should  threat  12  most  probably  be  assigned? 

8.  To  which  battery  should  threat  51  most  probably  be  assigned? 

9.  To  which  battery  should  threat  09  most  probably  be  assigned? 

10.  Which  threat  should  be  assigned  next? 

11.  What  is  the  overall  probability  of  kill  across  all  threats? 

12.  If  both  Philadelphia  computers  were  to  fail  now,  which  threat  would  you 
assign  first  to  the  Washington-Baltimore  complex? 

13.  If  both  Washington-Baltimore  computers  were  to  fail  now,  which  threat 
would  you  assign  first  to  the  Philadelphia  complex? 

14.  Which  battery(ies),  if  any,  selected  an  improper  missile  to  launch? 
(Answer  "None,  "  if  appropriate). 

15.  Assuming  a  wind  shift  of  180°,  for  which  threats  would  you  select  a  dif¬ 
ferent  missile  to  launch?  (Answer  "None,  "  if  appropriate). 

16.  Which  threat(s),  if  any,  should  be  assigned  to  more  than  one  battery? 
(Answer  "None,  "  if  appropriate). 


85 


QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATION  NO.  4 


1.  Which  weapon  should  be  launched  against  threat  09? 

2.  Which  weapon  should  be  launched  against  threat  57? 

3.  Which  weapon  should  be  launched  against  threat  51? 

4.  Which  weapon  should  be  launched  against  threat  55? 

5.  Which  weapon  should  be  launched  against  threat  11? 

6.  Which  weapon  should  be  launched  against  threat  56? 

7.  To  which  battery  should  threat  55  most  probably  be  assigned? 

8.  To  which  battery  should  threat  56  most  probably  be  assigned? 

9.  To  which  battery  should  threat  57  most  probably  be  assigned? 

10.  To  which  battery  should  threat  51  most  probably  be  assigned? 

11.  To  which  battery  should  threat  11  most  probably  be  assigned? 

12.  Which  threat  should  be  assigned  next? 

13.  What  is  the  overall  probability  of  kill  across  all  threats? 

14.  If  both  Philadelphia  computers  were  to  fail  now,  which  threat  would  you 
assign  first  to  the  Washington- Baltimore  complex? 

15.  If  both  Washington- Baltimore  computers  were  to  fail  now,  which  threat 
would  you  assign  first  to  the  Philadelphia  complex? 

16.  Which  battery(ies),  if  any,  selected  an  improper  missile  to  launch? 
(Answer  "None,  "  if  appropriate). 

17.  Assuming  a  wind  shift  of  180°,  for  which  threats  would  you  select  a  dif¬ 
ferent  missile  to  launch?  (Answer  "None,  "  if  appropriate). 

18.  Which  threat(s),  if  any,  should  be  assigned  to  more  than  one  battery? 
(Answer  "None,  "  if  appropriate). 
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QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATION  NO.  5 


1.  Which  weapon  should  be  launched  against  threat  08? 

2.  Which  weapon  should  be  launched  against  threat  13? 

3.  Which  weapon  should  be  launched  against  threat  05? 

4.  Which  weapon  should  be  launched  against  threat  09? 

5.  Which  weapon  should  be  launched  against  threat  55? 

6.  Which  weapon  should  be  launched  against  threat  01? 

7.  Which  weapon  should  be  launched  against  threat  54? 

8.  Which  weapon  should  be  launched  against  threat  12? 

9.  Which  weapon  should  be  launched  against  threai  52? 

10.  To  which  battery  should  threat  13  most  probably  be  assigned? 

11.  To  which  battery  should  threat  05  most  probably  be  assigned? 

12.  To  which  battery  should  threat  10  most  probably  be  assigned? 

13.  To  which  battery  should  threat  03  most  probably  be  assigned? 

14.  To  which  battery  should  threat  07  most  probably  be  assigned? 

15.  To  which  battery  should  threat  12  most  probably  be  assigned? 

16.  To  which  battery  should  threat  14  most  probably  be  assigned? 

17.  To  which  battery  should  threat  55  most  probably  be  assigned? 

18.  To  which  battery  should  threat  01  most  probably  be  assigned? 

19.  Which  threat  should  be  assigned  next? 

20.  What  is  the  overall  probability  of  kill  across  all  threats? 
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21.  If  both  Philadelphia  computers  were  to  fail  now,  which  threat  would  you 
assign  first  to  the  Washington- Baltimore  complex? 

22.  If  both  Washington- Baltimore  computers  were  to  fail  now,  which  threat 
would  you  assign  first  to  the  Philadelphia  complex? 

23.  Which  battery(ies),  if  any,  selected  an  improper  missile  to  launch? 
(Answer  "None,  "  if  appropriate). 

24.  Assuming  a  wind  shift  of  180°,  for  which  threats  would  you  select  a  dif¬ 
ferent  missile  to  launch?  (Answer  "None,  "  if  appropriate). 

25.  Which  threat(s),  if  any,  should  be  assigned  to  more  than  one  battery? 
(Answer  "None,  "  if  appropriate). 


88 


QUESTIONS  FOR  COMMAND  AND  CONTROL  SITUATION  NO.  6 


1.  Which  weapon  should  be  launched  against  threat  03? 

2.  Which  weapon  should  be  launched  against  threat  02? 

3.  Which  weapon  should  be  launched  against  threat  57? 

4.  Which  weapon  should  be  launched  against  threat  50? 

5.  Which  weapon  should  be  launched  against  threat  56? 

6.  Which  weapon  should  be  launched  against  threat  51? 

7.  Which  weapon  should  be  launched  against  threat  08? 

8.  To  which  battery  should  threat  56  most  probably  be  assigned? 

9.  To  which  battery  should  threat  57  most  probably  be  assigned? 

10.  To  which  battery  should  threat  02  most  probably  be  assigned? 

11.  To  which  battery  should  threat  08  most  probably  be  assigned? 

12.  To  which  battery  should  threat  03  most  probably  be  assigned? 

13.  Which  threat  should  be  assigned  next? 

14.  What  is  the  overall  probability  of  kill  across  all  threats? 

15.  If  both  Philadelphia  computers  were  to  fail  now,  which  threat  would  you 
assign  first  to  the  Washington- Baltimore  complex? 

16.  If  both  Washington- Baltimore  computers  were  to  fail  now,  which  threat 
would  you  assign  first  to  the  Philadelphia  complex? 

17.  Which  battery(ies),  if  any,  selected  an  improper  missile  to  launch? 
(Answer  "None,  "  if  appropriate). 

18.  Assuming  a  wind  shift  of  180°,  for  which  threats  would  you  select  a  dif¬ 
ferent  missile  to  launch?  (Answer  "None,  "  if  appropriate). 

19.  Which  threat(s),  if  any,  should  be  assigned  to  more  than  one  battery? 
(Answer  "None,  "  if  appropriate). 
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APPENDIX  F 

SAMPLE  DEI  CALCULATION 
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BATTERT  STATUS-PHILA 


FIGURE  F-l.  TRANSFER  CHART 


ABSOLUTE  ABSOLUTE  ABSOLUTE 

MISMATCH  L  INK  MISMATCH  LINK  MISMATCH  LINK 

LINK  01  SPLAY  ACTION  DIGITS  WEIGHTS  LINK  01  SPLAY  ACTION  DIGITS  WEIGHTS  LINK  DISPLAY  ACTION  DIGITS  WCIGHTS 

DIGITS  DIGITS  Lai  Lil  nq  .  DIGITS  DIGITS  _1&J _  _lB  l_  JtSU  SIGHS,  PI  SITS  _i«J -  — i  M I  ■ . 


—  O  O  O  O  O  -NN-OMOQ-OOOO  -OON-NO  —  <V  —  «V  MgMgo 


lllllmllllllllllmmlmlllllEi 


J\O0  J#0O  *MT\AW  bo  O  NV*V«  MOBOSO  MMNPMsO  «PVPy^WV*-iiN»  *<NBO  t\p'  »*\pm\ 
—  <rsjr  «-*  arvo  •  (TsffN/*- r  f'-f-f 


—  —  —IV  —  ——  —  <V  —  «V  —  *v— —  — iv  «v  •w  w 

-J 

< 


**VXU>"0«— OO  0^0 


—  iv  r— oo  <r\0  -  r-oo  <r\o  —  Mg  f\  i/>  ►- 

wsouaoAsosQsoss  <rs<r\<T'CTso  o  O  oo  o  u 

—  —  —  —  —  —  X 
O 
CD 


o  —  Mg  —  OOOOOVV—  O  -ON  -  <V  —  *V  —  Mgpg  —  NO  —  O  pg  <V  —  OOOOO 


KV»  PPNOO  "POO  oo  "PMPOO  "POO  tPNMNBO  f — oo  I - OO  BO  OO  MSKsW  HNWM—CO  M  «<V*S 

vt  « — r—  f— * oj»  arv  r— p~-^x  i/v^  ov»r-»v 


Mg  —  «V  «V  — <V  iV  —Mg  —  —  — «V  —Mg—  —  **>  —Mg 


r — oo  <r>®  —  Mg  kv»  1>*»  r— oo  <r»0  —  V  »Mvar  i/V-o  r  *oo  r>0  —  rg  rv>  r— oo  gnO  —  ig 


VVWOOOOO  —  MgOMgOOOOOOOOO  —  O  — O  —  <V  —  Mg  —  Mgpg  —  <V  — 


30  CO  Mr\PM\PM\«^M\0O  OO  KVO  -i  \  "MvTVT\30  IN  HM'lT.uPi  >»VlW»  o  l\0O  "M'OO  OO  oo  OO  M\» 
JN  JVIM5  Mg  Mg  rviiT'Ts'N  (TSCTVO  OVSffNiTV)  ffVrN*MNAiN-.  j»0  g»  (— TSOVTsd  f— 


—  — rgpg  —  —  “  —  —  —  —  V  —  Mg  ig  ig  WNryj  *m,  <v  —  —  ig 
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DEI  Worksheet 


1.  1  +  Z  (link  weights,  see  Transfer  Table)  =  1  +  224  =  225 

w 

2.  (n  +  m)  (number  of  used  indicators  and  decisions, 

u 

see  Transfer  Chart)  =  42 

3.  (n  +  m)^  (total  number  of  indicators  and  decisions, 

see  Transfer  Chart)  =  43 


4.  N  (total  number  of  links,  see  Transfer  Table) 

5.  Q  (total  number  of  indicator  and  decision  parts, 
see  Transfer  Chart) 

6.  nQ  (number  of  1  ) and  £>  ,  see  Transfer  Chart) 

7.  (Q  +  nQ) 

8.  N (n  +  m).  (Q  +  n  )  =  (105)  (43)  (273) 

t  o 


10.  Z  |m|  (absolute  mismatch  digits,  see  Transfer  Table) 
1  1.  1/4  Z  |m| 

12.  exp  (-  1/4  Z  |M]  )  | 


=  105 

=  235 
=  _38 
=  273 

=  1,  232,  595 
=  1,  1 10 
=  7.  24 
=  1. 81 
=  .  164 


(n  +  m)^  [  exp  (-  1/4  Z  |M|  )  ] 

(1  +  £  )  [  \/  N  (n  +  m).  (Q  +  n 
w  t  o 


(42)  (.  164) 
(225)  (1110) 


6.  888 

249,  750 


=  .  0000276  x  10,  000 

=  .  276 
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